Tag
LLaDA-Image presents a unified framework that combines a 6B diffusion transformer with a frozen vision-language module for generating photorealistic images with precise editing, achieving state-of-the-art results among open-source models through efficient training and fast inference.
This paper analyzes how finite Newton-Schulz iterations in the Muon optimizer benefit nonsmooth nonconvex optimization by smoothing the polar map, providing convergence guarantees that match best-known bounds.
This paper scales the Muon optimizer for Diffusion Transformers from 1.3B to 15B parameters, introducing Periodic Row-wise Muon to reduce computational overhead while preserving generative quality improvements over AdamW.
This paper proposes FedCoMuon, a federated compositional Muon optimizer for matrix-wise models, along with a variance-reduced variant (FedCoMuon-VR). The authors provide convergence analysis under non-i.i.d. and non-convex settings, showing improved sample complexity over existing FedMuon algorithms, and demonstrate competitive performance on robust federated learning and task-distributed risk-sensitive meta learning.
This paper ablates the Muon optimizer to find that orthogonalization (Newton-Schulz iteration), not spectral scaling, is the key ingredient behind its faster grokking on modular arithmetic, and introduces a stability-aware metric for measuring grokking speed.
This paper studies how much orthogonalization the Muon optimizer requires, proposing a five-step cubic Newton-Schulz schedule that reduces computational cost while achieving training quality similar to more expensive methods across GPT-2 Small and hybrid MoE/Mamba models.
Discussion of a finding that all softmax/linear attention variants can be interpolated, and that the Muon optimizer is crucial for Parallax to move beyond Softmax Attention. Includes link to paper and code.
SignMuon is a 1-bit, matrix-aware optimizer for distributed training that combines signSGD's majority-vote sign aggregation with Muon's polar-step framework, achieving 32x bandwidth reduction over float32 while maintaining strong convergence and performance on benchmarks like CIFAR-10/ResNet-50 and nanoGPT.