heavy-tailed-noise

This paper establishes the first convergence guarantees for the plain vector-form Adam optimizer under heavy-tailed stochastic noise, showing it converges to stationary points but with suboptimal iteration complexity, and improves to optimal rates with a known domain radius.

This paper provides the first comprehensive convergence analysis of vanilla SGD with momentum under heavy-tailed noise without gradient clipping or normalization, revealing inferior rates compared to clipped variants and supported by experiments on synthetic functions.