Tag
The paper provides an asymptotic theory explaining why Muon's orthogonalization preconditioner matches the neural network Hessian structure at initialization, showing that overparameterization drives diagonal dominance of E[GG^T] across four canonical models.
This paper develops a statistical theory of grokking in high-dimensional regression, showing how regularization geometry and signal sparsity govern generalization near interpolation, including a zero-one generalization law and instability of minimum-norm interpolators.
This paper proposes performing PAC-Bayesian analysis on quotient parameter spaces to remove KL contributions from parameter symmetries, and constructs a geometry-induced prior that approximates the ideal posterior-matched prior, resulting in tighter generalization bounds. Experiments on Fourier regression and Query-Key attention show significant reductions in KL divergence and certificate values.
This paper revisits the volume hypothesis, which posits that generalization in over-parameterized networks is mainly due to the larger volume of good-generalizing regions in weight space rather than SGD's implicit bias. Through experiments with binary networks, the authors show that the generalization advantage of gradient learning over random sampling diminishes as training data size grows, potentially resolving contradictory prior findings.
A paper investigating the reasons behind the success of overparameterization in neural networks, comparing the lottery ticket hypothesis with escape dimensions.
OpenAI research reveals the 'double descent' phenomenon where test error exhibits a non-monotonic pattern as both model size and training steps increase, challenging traditional understanding of the bias-variance tradeoff in deep learning.