Tag
Introduces Sharpness-Guided Equilibrium Sampling (SGS) that dynamically adjusts sampling probabilities using sharpness estimates to achieve balanced flat minima in long-tailed learning, achieving significant gains on CIFAR-100 LT and ImageNet-LT.
The paper proposes GEAR-SAM, which adaptively allocates the perturbation budget across network blocks using an exponential moving average of squared gradients, improving generalization without additional computational overhead.
Proposes EISAM, a new optimizer that extends Sharpness-Aware Minimization using an extragradient step to find flatter minima, improving generalization and robustness while reducing sensitivity to hyperparameters. Outperforms SGD, Adam, and SAM on benchmarks.
This paper challenges the common belief that flat minima cause better generalization in neural networks, arguing that 'weakness'—a reparameterization-invariant measure of function simplicity—is the true driver. Empirical results on MNIST and Fashion-MNIST show that weakness predicts generalization while sharpness anticorrelates, and the large-batch generalization advantage vanishes as training data increases.