Tag
This paper establishes convergence guarantees for gradient descent on general feedforward neural networks of arbitrary width/depth, using a novel generalized Lipschitz smoothness condition that holds for common activations and mean-squared error, without special initialization or dataset requirements.
This thesis tackles seven challenges in distributed and federated optimization, introducing methods like ProxSkip and Variance Reduced ProxSkip, and establishing theoretical foundations for communication-efficient, robust, and practical algorithms.
This theoretical paper studies the non-asymptotic implicit bias of logistic regression under gradient descent, proving that the parameter vector weakly aligns with the max-margin direction quickly, within a doubly exponential number of iterations in the alignment error.
This paper proposes an adaptive gradient descent method using one-sided Hölder regularity to control step sizes based on directional curvature rather than full gradient variation, providing convergence guarantees for nonconvex objectives and demonstrating empirical benefits.
This paper empirically validates that the quadratic model of optimization can predict pretraining dynamics in LLMs up to 150M parameters, analyzing Hessian spectrum and local stability.
This article argues that specialization is inevitable for AI systems, drawing on evidence from optimization theory, evolutionary biology, competitive markets, and machine learning. It interprets a 2026 paper by Goldfeder, Wyder, LeCun, and Shwartz-Ziv to challenge the assumption that greater capability leads to greater generality.
This paper analyzes generalization error, uniform stability, and uniform argument stability of gradient descent (GD) and stochastic gradient descent (SGD) over discrete parameter spaces with deterministic or stochastic rounding, showing that rounding degrades generalization for GD and introduces dimension-dependent errors for stochastic rounding.
This paper presents a unified theoretical framework for stochastic variance-reduced estimation, deriving high-probability bounds via a new Freedman inequality and improving oracle complexities for constrained optimization.
GPT-5 helped mathematician Ernest Ryu solve a 40-year-old open problem in optimization theory regarding the Nesterov Accelerated Gradient method's stability properties. The breakthrough demonstrates LLMs' capability to assist in significant mathematical discovery by surfacing relevant techniques and ideas from across mathematical literature.