Tag
This paper introduces a trust-region framework for understanding adaptive moment estimation methods like Adam, deriving a family of learning-rate mechanisms based on p-th moment constraints (2≤p≤4), including a kurtosis-based variant (Gmake). Experiments on GPT2-124M show that second-moment versions become competitive when trust-region controls are stronger.
The paper proposes a parallel architecture that assembles static gradient methods to achieve adaptivity in stochastic gradient descent, simplifying convergence analysis while retaining parameter adaptivity.
This paper sharpens the convergence analysis of LoRA, improving deterministic oracle complexity from exponential to O(epsilon^{-4}), and proposes stochastic variants LoRA-NSGDM and LoRA-STORM with improved oracle complexities of O(epsilon^{-8}) and O(epsilon^{-6}) respectively.
This paper proposes new discrete-time approximations for stochastic gradient Langevin dynamics (SGLD) with and without momentum, enabling accurate predictions of stationary covariance, iterate average covariance, and integrated autocorrelation time. The method provides improved tuning guidance for large-sample uncertainty quantification, especially under model misspecification.
This paper proposes LMO-IGT, a new class of stochastic optimization methods that accelerates convergence using implicit gradient transport while maintaining a single-gradient-per-iteration structure. It introduces a unified theoretical framework and demonstrates improved performance over existing LMO-based optimizers like Muon.