Tag
The paper proposes Ori, a drift-aware streaming k-PCA low-rank optimizer for LLM training inspired by Oja++, which tracks shifting principal gradient directions at low per-step cost, reducing the performance gap to full-rank optimizers by up to 69.3% on 1B LLaMA pretraining and achieving best performance in 10 of 16 RoBERTa-Base fine-tuning scenarios.
A mechanistic study of emergent misalignment in LLMs finds that directional Hessian curvature concentrates on semantic pivot tokens and that harmful-safesubspace divergence drives failures; the authors propose a geometric mitigation framework that orthogonally projects harmful gradient subspaces, suppressing emergent misalignment by up to 80% on Qwen2.5-14B-IT.
This paper empirically shows that the gradient's top-r subspace in low-rank training methods like GaLore is non-identifiable beyond a small reproducible core, with estimator noise dominating apparent rotations. It analyzes the implications for optimizer state transport and introduces LDAdam, which outperforms GaLore in perplexity.