Tag
Dion3 is a revised Muon optimizer that reduces computational and communication overhead via Gram Newton-Schulz, symmetric GEMM kernels, and megabatching, achieving up to 6x faster optimizer steps while matching or improving loss.
This paper ablates the Muon optimizer to find that orthogonalization (Newton-Schulz iteration), not spectral scaling, is the key ingredient behind its faster grokking on modular arithmetic, and introduces a stability-aware metric for measuring grokking speed.
This work proposes orthogonalizing the memory matrix of mLSTM recurrent models to improve their performance on noisy associative recall tasks. Experiments show that using Newton-Schulz iterations for read-only orthogonalization enhances validation accuracy compared to baseline mLSTM.
This paper proposes SCBoost, a boosting framework that reduces learner redundancy by projecting residuals onto the orthogonal complement of previous predictions and using covariance-regularized weighting, with theoretical guarantees and strong empirical performance.
This blog post presents Gram Newton-Schulz, a hardware-aware optimization of the Newton-Schulz orthogonalization procedure used in the Muon optimizer, achieving significant speedups for training large language models while preserving model quality.
This paper proposes Dynamic Contextual Orthogonalization (DCO), an inference-time method that reduces hallucinations in large language models by aligning attention head outputs with the context manifold, achieving superior faithfulness on benchmarks with Llama-3 models.
This paper studies how much orthogonalization the Muon optimizer requires, proposing a five-step cubic Newton-Schulz schedule that reduces computational cost while achieving training quality similar to more expensive methods across GPT-2 Small and hybrid MoE/Mamba models.