Tag
This paper explains why increasing batch size during LLM pretraining helps, by proposing a generalized Blum–Gladyshev noise model with a tunable variance growth exponent, deriving matching information-theoretic oracle complexity bounds, and introducing an adaptive batch scheduler that achieves lower validation loss on OLMo2 models up to 1B parameters while using under 10% of small-batch iterations.
论文研究了在未知重尾噪声统计下在线凸优化的区间动态遗憾界，提出一个不依赖 G、σ、p、区间长度等参数的学习器，并证明了包含噪声幂指数的通用遗憾上界及匹配的下界。
The paper introduces a federated stochastic bilevel optimization algorithm that uses only first-order gradients to avoid second-order matrix computations, reducing running time, and includes a novel learning rate mechanism with experimental confirmation.
The paper analyzes the delayed effects of minibatch perturbations in AdamW by modeling it as a finite-horizon input-state-output system, revealing how optimizer states influence training dynamics.
This paper establishes the first convergence guarantees for the plain vector-form Adam optimizer under heavy-tailed stochastic noise, showing it converges to stationary points but with suboptimal iteration complexity, and improves to optimal rates with a known domain radius.
This paper introduces a framework to reduce per-sample harm in stochastic optimization, where parameter updates from batch averaging and historical states increase individual sample loss. The method uses dimensionality reduction and focuses on the last linear layer for efficiency, showing improved generalization on image classification tasks.
This paper proposes a regularity-aware stochastic multi-gradient descent method (MoRe) that adaptively switches between conflict-avoidant and scalarization updates. The method achieves improved convergence rates from O~T^{-1/4} to O~T^{-1/2} in nonconvex settings while maintaining per-iterate conflict avoidance.
Proposes MGUP, a momentum-gradient alignment update policy for selective intra-layer parameter updates in stochastic optimization, which integrates with optimizers like AdamW, Lion, and Muon, and provides theoretical convergence guarantees along with superior performance on large-scale model training tasks.
This paper solves a COLT open problem by providing an optimal gap-dependent regret algorithm for private stochastic decision-theoretic online learning, achieving the lower bound of order (log K)/Δ_min + (log K)/ε.
This paper presents a unified theoretical framework for stochastic variance-reduced estimation, deriving high-probability bounds via a new Freedman inequality and improving oracle complexities for constrained optimization.
This paper studies nonconvex stochastic optimization under Blum-Gladyshev noise, where gradient variance grows with distance from initialization. It proves convergence guarantees for normalized SGD with momentum and a variance-reduced STORM method, achieving minimax optimal rates under certain conditions.