Tag
This paper analyzes Adam as a diagonal empirical Fisher approximation to natural gradient descent, measuring its geometric deviation across linear, logistic, and small neural network loss landscapes. Results show deviation grows under ill-conditioning but Adam still reaches low loss, suggesting its effectiveness comes from approximation-error balance and momentum smoothing rather than true NGD tracking.