Tag
The paper introduces RODE, an optimizer that decouples radial and directional components for matrix updates, demonstrating superior performance over Muon variants in language modeling and image classification tasks.
Zeta proposes a dual whitening optimizer that applies coordinate whitening before spectral whitening to resolve scale heterogeneity in momentum matrices, reducing orthogonalization error and improving convergence and generalization in large-scale neural network training.