Tag
This paper analyzes when and how the Muon optimizer provides preconditioning benefits, showing it offers no acceleration under isotropic noise but can reduce task-frequency dependence from linear to square-root under structured mini-batch gradient noise, while avoiding SignSGD's rotation-induced negative transfer.
The paper provides an asymptotic theory explaining why Muon's orthogonalization preconditioner matches the neural network Hessian structure at initialization, showing that overparameterization drives diagonal dominance of E[GG^T] across four canonical models.
A paper examining a single-rollout asynchronous RL recipe combining the Muon optimizer, token-averaged loss, truncated importance sampling, and auxiliary penalties for training long-horizon repository-level issue resolution agents, showing improved training success over baselines across two model families.
Temperon is a training method that uses Sharpness-aware minimization (SAM) only in the final phase of training to achieve full SAM quality with a third less wall-clock time, validated on vision and language tasks.
FedSubMuon is a communication-efficient federated fine-tuning method for LLMs that optimizes compact coefficient matrices within structured subspaces to reduce upload costs while maintaining strong performance.
LLaDA-Image presents a unified framework that combines a 6B diffusion transformer with a frozen vision-language module for generating photorealistic images with precise editing, achieving state-of-the-art results among open-source models through efficient training and fast inference.
This paper analyzes how finite Newton-Schulz iterations in the Muon optimizer benefit nonsmooth nonconvex optimization by smoothing the polar map, providing convergence guarantees that match best-known bounds.
This paper scales the Muon optimizer for Diffusion Transformers from 1.3B to 15B parameters, introducing Periodic Row-wise Muon to reduce computational overhead while preserving generative quality improvements over AdamW.
This paper proposes FedCoMuon, a federated compositional Muon optimizer for matrix-wise models, along with a variance-reduced variant (FedCoMuon-VR). The authors provide convergence analysis under non-i.i.d. and non-convex settings, showing improved sample complexity over existing FedMuon algorithms, and demonstrate competitive performance on robust federated learning and task-distributed risk-sensitive meta learning.
This paper ablates the Muon optimizer to find that orthogonalization (Newton-Schulz iteration), not spectral scaling, is the key ingredient behind its faster grokking on modular arithmetic, and introduces a stability-aware metric for measuring grokking speed.
This paper studies how much orthogonalization the Muon optimizer requires, proposing a five-step cubic Newton-Schulz schedule that reduces computational cost while achieving training quality similar to more expensive methods across GPT-2 Small and hybrid MoE/Mamba models.
Discussion of a finding that all softmax/linear attention variants can be interpolated, and that the Muon optimizer is crucial for Parallax to move beyond Softmax Attention. Includes link to paper and code.
SignMuon is a 1-bit, matrix-aware optimizer for distributed training that combines signSGD's majority-vote sign aggregation with Muon's polar-step framework, achieving 32x bandwidth reduction over float32 while maintaining strong convergence and performance on benchmarks like CIFAR-10/ResNet-50 and nanoGPT.