sgd

This article discusses a research paper showing that the disagreement rate between two deep networks trained with different random seeds can accurately estimate generalization error using only unlabeled data, revealing a surprising connection called Generalization Disagreement Equality.

This paper provably shows that whether learning-rate cooldown helps in WSD schedules depends on the structure of gradient noise and whether the optimizer normalizes its update, explaining why cooldown can be ineffective for SGD but necessary for normalized methods.

This paper provides optimal high-probability bounds for stochastic gradient descent under Markovian noise for PL-smooth objectives, closing gaps between expectation and high-probability guarantees and extending to heavy-tailed settings with matching lower bounds.

This paper derives batch scaling laws for sketched linear regression under power-law spectra, analyzing one-pass and multi-pass mini-batch SGD. It provides explicit risk decompositions showing how batch size affects bias, variance, and fluctuation terms, and establishes that without-replacement sampling yields lower noise than with-replacement.

This paper establishes the first population risk bounds for Kolmogorov-Arnold Networks trained with mini-batch SGD and DP-SGD using correlated noise, advancing theoretical understanding of KANs in privacy-sensitive domains.

This paper challenges the geometric justification for the Muon optimizer, arguing that precise structure is less important than step-size optimality. It introduces Freon and Kaon optimizers to demonstrate that random or inverted spectra can perform as well as Muon.