@TensorTonic: 7 math ideas every ML engineer uses daily and almost nobody has actually derived: 1. Why gradient descent moves in the …

X AI KOLs Timeline News
machine-learning math gradient-descent backpropagation kl-divergence adam-optimizer eigenvectors

Summary

This tweet lists 7 fundamental math ideas used daily by ML engineers, with brief explanations emphasizing the underlying derivations, such as why gradient descent moves in the steepest direction and why softmax plus cross-entropy yields a clean gradient.

7 math ideas every ML engineer uses daily and almost nobody has actually derived: 1. Why gradient descent moves in the direction of steepest descent, not just downhill, but provably the steepest direction, straight from the definition of a directional derivative. 2. Why softmax plus cross-entropy collapses into that suspiciously clean gradient of pred minus true, and what breaks the moment you swap the loss function. 3. Why the chain rule is backprop, not an analogy for it, the same operation applied mechanically to a computation graph. 4. Why dividing attention scores by root d_k isn't arbitrary, it's variance control, derivable from how dot products scale with dimension. 5. Why KL divergence isn't symmetric, and what that asymmetry actually costs you when you pick forward vs reverse KL. 6. Why Adam's second moment estimate quietly approximates a diagonal Hessian, making it quasi-Newton in disguise. 7. Why eigenvectors are the directions a matrix doesn't rotate, the one geometric fact that makes SVD, PCA, and spectral clustering all click at once.
Original Article
View Cached Full Text

Cached at: 07/11/26, 11:24 AM

7 math ideas every ML engineer uses daily and almost nobody has actually derived:

  1. Why gradient descent moves in the direction of steepest descent, not just downhill, but provably the steepest direction, straight from the definition of a directional derivative.

  2. Why softmax plus cross-entropy collapses into that suspiciously clean gradient of pred minus true, and what breaks the moment you swap the loss function.

  3. Why the chain rule is backprop, not an analogy for it, the same operation applied mechanically to a computation graph.

  4. Why dividing attention scores by root d_k isn’t arbitrary, it’s variance control, derivable from how dot products scale with dimension.

  5. Why KL divergence isn’t symmetric, and what that asymmetry actually costs you when you pick forward vs reverse KL.

  6. Why Adam’s second moment estimate quietly approximates a diagonal Hessian, making it quasi-Newton in disguise.

  7. Why eigenvectors are the directions a matrix doesn’t rotate, the one geometric fact that makes SVD, PCA, and spectral clustering all click at once.

Similar Articles