orthogonal-gradients

This paper introduces OrthoGrad, a geometric intervention that removes the radial component of weight gradients during optimization, and shows that it reduces memorization of noisy labels in small-data regimes but does not prevent eventual memorization.