Tag
This paper provides the first comprehensive convergence analysis of vanilla SGD with momentum under heavy-tailed noise without gradient clipping or normalization, revealing inferior rates compared to clipped variants and supported by experiments on synthetic functions.
The paper proposes FedCGNM, a client-side optimizer that groups classes and normalizes momentum per group to address class imbalance in federated learning, along with FedHOO for efficient hyperparameter exploration. Empirical results show consistent improvements over baselines.
A University of Johannesburg working paper proposes a Kalman-filtered, momentum-extended Anticor algorithm (K-ACM) that, in a backtest from 1962–1984, produces an extreme return of $1 to $36 billion, though acknowledged as a backtest artifact due to unrealistic assumptions.
Proposes MGUP, a momentum-gradient alignment update policy for selective intra-layer parameter updates in stochastic optimization, which integrates with optimizers like AdamW, Lion, and Muon, and provides theoretical convergence guarantees along with superior performance on large-scale model training tasks.
DP-MacAdam combines adaptive clipping and adaptive momentum to improve differentially private SGD, achieving better model utility without manual tuning of the clipping threshold.