adam

This paper establishes the first convergence guarantees for the plain vector-form Adam optimizer under heavy-tailed stochastic noise, showing it converges to stationary points but with suboptimal iteration complexity, and improves to optimal rates with a known domain radius.

Introduces PsiLogic, a chaos-aware optimizer that augments Adam with a dynamic damping term based on gradient instability, and proposes FairBench for reproducible evaluation. Shows competitive or superior results on NLP, ViT, and ResNet tasks with full transparency on limitations.

This paper empirically shows that the gradient's top-r subspace in low-rank training methods like GaLore is non-identifiable beyond a small reproducible core, with estimator noise dominating apparent rotations. It analyzes the implications for optimizer state transport and introduces LDAdam, which outperforms GaLore in perplexity.

This paper establishes a general theory of the Adam optimizer for time-varying and nonstationary stochastic systems, providing parameter tracking and output prediction error bounds under a stochastic excitation condition that allows nonstationary and dependent data.

This paper investigates why the Muon optimizer outperforms Adam in large language model training, showing from a curvature perspective that Muon incurs a smaller curvature penalty due to lower normalized directional sharpness, with advantages amplified by data imbalance.

This paper generalizes non-uniform smoothness assumptions to objectives whose curvature is affine in the objective value, proving convergence rates for steepest descent and diagonal variants of RMSProp and Adam, with applications to logistic regression and neural networks.

This paper introduces a 'rod flow' model for Adam and other adaptive optimizers to better analyze their behavior at the edge of stability. It extends continuous-time modeling to momentum methods, showing improved accuracy in tracking discrete iterates compared to stable flow models.

This paper revisits the Adam optimizer for streaming reinforcement learning, demonstrating that established methods like DQN and C51 perform well when properly tuned. The authors propose Adaptive Q(lambda), which combines eligibility traces with Adam's variance adaptation to surpass existing streaming RL methods on 55 Atari games.

Research paper investigating performance degradation when using the Muon optimizer instead of Adam for fine-tuning pretrained models, demonstrating that parameter-efficient methods like LoRA effectively mitigate this optimizer mismatch across language and vision tasks.