low-rank-training

This paper empirically shows that the gradient's top-r subspace in low-rank training methods like GaLore is non-identifiable beyond a small reproducible core, with estimator noise dominating apparent rotations. It analyzes the implications for optimizer state transport and introduces LDAdam, which outperforms GaLore in perplexity.