@0xLogicrw: Tilde Research found a hidden flaw in the Muon optimizer, used by leading models like DeepSeek V4, Kimi K2.5, and GLM-5: it causes over a quarter of MLP layer neurons to die permanently in early training. The team designed an alternative optimizer, Auro…

Tilde Research discovered a hidden flaw in the Muon optimizer, utilized by leading models such as DeepSeek V4, Kimi K2.5, and GLM-5: it causes more than a quarter of the neurons in MLP layers to die permanently during the early stages of training. Based on this finding, the team designed and open-sourced an alternative optimizer called Aurora. A 1.1B model trained with only about 100B tokens matched the performance of Qwen3-1.7B (trained on 36T tokens) on language understanding benchmarks like HellaSwag and Winogrande. The issue stems from a mathematical characteristic of how Muon handles MLP weight matrices. In the early stages of training, some neurons happen to receive weaker gradient signals. Traditional optimizers like AdamW normalize per parameter, naturally flattening these disparities; however, Muon's orthogonalization step passes these weak signals through unchanged. Weak neurons continue to receive weak updates, becoming increasingly silent, creating a "rich get richer" vicious cycle. By step 500 of training, over a quarter of the neurons had substantially died, wasting parameter capacity. The previous improved version, NorMuon, alleviated this by forcing the flattening of update magnitudes per row, but at the cost of breaking the orthogonality of the update matrix (orthogonalization ensures each update step is as efficient as possible, which is a core advantage of Muon), resulting in a loss of optimization precision. Aurora treats "uniform updates" and "orthogonality" as joint constraints, using alternating iterations to satisfy both simultaneously: ensuring every neuron gets a fair chance to learn without sacrificing update precision. Untuned Aurora incurs only a 6% higher computational overhead than Muon and can be directly substituted. In the modded-nanoGPT optimization benchmark, Aurora set a new best record at 3175 steps. Aurora's advantages amplify with increasing MLP width; the higher the expansion coefficient, the more pronounced the improvement. The code and the 1.1B pre-trained model are both open-source.