Tag
The paper proposes Ori, a drift-aware streaming k-PCA low-rank optimizer for LLM training inspired by Oja++, which tracks shifting principal gradient directions at low per-step cost, reducing the performance gap to full-rank optimizers by up to 69.3% on 1B LLaMA pretraining and achieving best performance in 10 of 16 RoBERTa-Base fine-tuning scenarios.