Tag
This paper establishes the first convergence guarantees for the plain vector-form Adam optimizer under heavy-tailed stochastic noise, showing it converges to stationary points but with suboptimal iteration complexity, and improves to optimal rates with a known domain radius.
This paper proposes an adaptive gradient descent method using one-sided Hölder regularity to control step sizes based on directional curvature rather than full gradient variation, providing convergence guarantees for nonconvex objectives and demonstrating empirical benefits.
This paper proves that online gradient descent achieves optimal √T regret for hidden-convex losses under a Hessian compatibility condition, resolving open questions in adversarial online learning. It also extends results to one-point bandit feedback with a T^{3/4} expected regret bound.
This paper proposes a robust subspace-constrained quadratic model for learning low-dimensional structures from high-dimensional data, accommodating heavy-tailed noise. A gradient-based algorithm with backtracking line search is developed, and experiments show improved robustness and reconstruction accuracy.
This paper studies nonconvex stochastic optimization under Blum-Gladyshev noise, where gradient variance grows with distance from initialization. It proves convergence guarantees for normalized SGD with momentum and a variance-reduced STORM method, achieving minimax optimal rates under certain conditions.
This paper introduces SHAPE, a structured adaptive port-Hamiltonian optimizer for fixed-budget nonconvex optimization that uses event-triggered mechanisms to balance descent, exploration, and budget allocation.