Tag
This paper proposes performing PAC-Bayesian analysis on quotient parameter spaces to remove KL contributions from parameter symmetries, and constructs a geometry-induced prior that approximates the ideal posterior-matched prior, resulting in tighter generalization bounds. Experiments on Fourier regression and Query-Key attention show significant reductions in KL divergence and certificate values.
This paper revisits the volume hypothesis, which posits that generalization in over-parameterized networks is mainly due to the larger volume of good-generalizing regions in weight space rather than SGD's implicit bias. Through experiments with binary networks, the authors show that the generalization advantage of gradient learning over random sampling diminishes as training data size grows, potentially resolving contradictory prior findings.
A paper investigating the reasons behind the success of overparameterization in neural networks, comparing the lottery ticket hypothesis with escape dimensions.
OpenAI research reveals the 'double descent' phenomenon where test error exhibits a non-monotonic pattern as both model size and training steps increase, challenging traditional understanding of the bias-variance tradeoff in deep learning.