preconditioning

SS-eSOAP introduces a self-scaled adaptive preconditioning method for physics-informed neural networks, improving optimization by reducing residuals and achieving higher accuracy on various PDE benchmarks compared to baseline methods.

This paper investigates why Adam does not exhibit gradient descent's implicit low-rank bias in factored models, showing that coordinate-wise preconditioning breaks the relevant symmetry, while shared-scalar methods like Muon and Shampoo preserve it.

This paper update presents a universal sequence preconditioning method achieving dimension-free regret bounds for marginally stable linear dynamical systems, using second-order VAW algorithm and Faber polynomials.

Zeta proposes a dual whitening optimizer that applies coordinate whitening before spectral whitening to resolve scale heterogeneity in momentum matrices, reducing orthogonalization error and improving convergence and generalization in large-scale neural network training.

This paper proposes a reparametrization of the preconditioner in Shampoo-based optimization methods (like KL-Shampoo and SOAP) to support BFloat16 storage and reduce computational overhead by updating only part of the basis via QR decomposition in a subspace, making these methods more memory- and time-efficient.

AdaPreLoRA is a novel LoRA optimizer that uses Adafactor diagonal Kronecker preconditioning to improve factor-space updates while maintaining low memory usage, demonstrating competitive performance across various LLMs and tasks.