adam-optimizer

This paper provides a theoretical analysis of the edge-of-stability phenomenon for the Adam optimizer on a one-dimensional quadratic function, proving that Adam exhibits a restoring mechanism that pushes sharpness toward a stability threshold.

This tweet lists 7 fundamental math ideas used daily by ML engineers, with brief explanations emphasizing the underlying derivations, such as why gradient descent moves in the steepest direction and why softmax plus cross-entropy yields a clean gradient.