Tag
This paper introduces a framework to reduce per-sample harm in stochastic optimization, where parameter updates from batch averaging and historical states increase individual sample loss. The method uses dimensionality reduction and focuses on the last linear layer for efficiency, showing improved generalization on image classification tasks.
This paper proposes a regularity-aware stochastic multi-gradient descent method (MoRe) that adaptively switches between conflict-avoidant and scalarization updates. The method achieves improved convergence rates from O~T^{-1/4} to O~T^{-1/2} in nonconvex settings while maintaining per-iterate conflict avoidance.
Proposes MGUP, a momentum-gradient alignment update policy for selective intra-layer parameter updates in stochastic optimization, which integrates with optimizers like AdamW, Lion, and Muon, and provides theoretical convergence guarantees along with superior performance on large-scale model training tasks.
This paper solves a COLT open problem by providing an optimal gap-dependent regret algorithm for private stochastic decision-theoretic online learning, achieving the lower bound of order (log K)/Δ_min + (log K)/ε.
This paper presents a unified theoretical framework for stochastic variance-reduced estimation, deriving high-probability bounds via a new Freedman inequality and improving oracle complexities for constrained optimization.
This paper studies nonconvex stochastic optimization under Blum-Gladyshev noise, where gradient variance grows with distance from initialization. It proves convergence guarantees for normalized SGD with momentum and a variance-reduced STORM method, achieving minimax optimal rates under certain conditions.