low-rank-bias

This paper investigates why Adam does not exhibit gradient descent's implicit low-rank bias in factored models, showing that coordinate-wise preconditioning breaks the relevant symmetry, while shared-scalar methods like Muon and Shampoo preserve it.

This paper studies how depth alone induces an implicit low-rank bias in deep unconstrained feature models trained without regularization, shifting the optimal solution from neural collapse to softmax codes, and provides the first asymptotic and dynamic characterization of this bias under gradient descent with cross-entropy loss.