Tag
This paper examines the coupling between the preconditioning exponent and learning rate in adaptive optimizers, showing that the exponent maximizing cross-environment accuracy decreases with log learning rate, and identifies a conflict between source-domain validation and robustness to environmental change.
This paper presents Iso-LoRA, an optimizer that enhances LoRA by promoting even energy distribution across singular directions through spectral descent on tangent-space perturbations, improving effective rank and downstream performance in language model adaptation.
Introduces a symmetry-compatible principle for LLM optimizer design, yielding a layerwise optimizer stack with principled updates for embeddings, LM heads, SwiGLU MLPs, and MoE routers, showing improved validation loss over AdamW across multiple architectures.