Tag
The paper explores whether any neural network can be redesigned to train effectively with gradient descent, proving a universality result that for any network, there exists an extension that reproduces given weights and outputs via gradient descent.
Forward-Pass-Only MLP training (FPO) adapts large language models without backpropagation, achieving 2.7–3.2× higher throughput and 40% less peak memory while maintaining benchmark performance.
Aurora is a leverage-aware spectral optimizer that addresses neuron death in MLP layers by enforcing row uniformity while preserving the polar factor geometry of Muon updates, achieving state-of-the-art performance on the modded-nanoGPT speedrun benchmark.
Zeta proposes a dual whitening optimizer that applies coordinate whitening before spectral whitening to resolve scale heterogeneity in momentum matrices, reducing orthogonalization error and improving convergence and generalization in large-scale neural network training.
This paper introduces Muon^p, a novel optimizer that uses fractional spectral-power updates to interpolate between Muon and gradient descent, providing theoretical justification and empirical gains on billion-scale fine-tuning tasks.
This paper studies how much orthogonalization the Muon optimizer requires, proposing a five-step cubic Newton-Schulz schedule that reduces computational cost while achieving training quality similar to more expensive methods across GPT-2 Small and hybrid MoE/Mamba models.