Tag
Commentary on the Marin 535B-A23B technical report: this 535B total / 23B active MoE model was trained to completion on 18T tokens. The report clearly documents the real decision-making process behind large-scale training, component trade-offs, and risk management. Through small-scale experiments and a scaling ladder, loss was predicted in advance, and the final run outperformed predictions while remaining stable throughout.
Meta and Mila researchers show that temporary weight decay during Muon-pretrained Transformers can appear harmful at removal yet improve final loss, demonstrating delayed effects driven by persistent weight-norm deficits that should be evaluated at the full training budget.
This paper analyzes when and how the Muon optimizer provides preconditioning benefits, showing it offers no acceleration under isotropic noise but can reduce task-frequency dependence from linear to square-root under structured mini-batch gradient noise, while avoiding SignSGD's rotation-induced negative transfer.
The paper provides an asymptotic theory explaining why Muon's orthogonalization preconditioner matches the neural network Hessian structure at initialization, showing that overparameterization drives diagonal dominance of E[GG^T] across four canonical models.
A paper examining a single-rollout asynchronous RL recipe combining the Muon optimizer, token-averaged loss, truncated importance sampling, and auxiliary penalties for training long-horizon repository-level issue resolution agents, showing improved training success over baselines across two model families.
Temperon is a training method that uses Sharpness-aware minimization (SAM) only in the final phase of training to achieve full SAM quality with a third less wall-clock time, validated on vision and language tasks.
FedSubMuon is a communication-efficient federated fine-tuning method for LLMs that optimizes compact coefficient matrices within structured subspaces to reduce upload costs while maintaining strong performance.
LLaDA-Image presents a unified framework that combines a 6B diffusion transformer with a frozen vision-language module for generating photorealistic images with precise editing, achieving state-of-the-art results among open-source models through efficient training and fast inference.
This paper analyzes how finite Newton-Schulz iterations in the Muon optimizer benefit nonsmooth nonconvex optimization by smoothing the polar map, providing convergence guarantees that match best-known bounds.
This paper scales the Muon optimizer for Diffusion Transformers from 1.3B to 15B parameters, introducing Periodic Row-wise Muon to reduce computational overhead while preserving generative quality improvements over AdamW.
This paper proposes FedCoMuon, a federated compositional Muon optimizer for matrix-wise models, along with a variance-reduced variant (FedCoMuon-VR). The authors provide convergence analysis under non-i.i.d. and non-convex settings, showing improved sample complexity over existing FedMuon algorithms, and demonstrate competitive performance on robust federated learning and task-distributed risk-sensitive meta learning.
This paper ablates the Muon optimizer to find that orthogonalization (Newton-Schulz iteration), not spectral scaling, is the key ingredient behind its faster grokking on modular arithmetic, and introduces a stability-aware metric for measuring grokking speed.
This paper studies how much orthogonalization the Muon optimizer requires, proposing a five-step cubic Newton-Schulz schedule that reduces computational cost while achieving training quality similar to more expensive methods across GPT-2 Small and hybrid MoE/Mamba models.
Discussion of a finding that all softmax/linear attention variants can be interpolated, and that the Muon optimizer is crucial for Parallax to move beyond Softmax Attention. Includes link to paper and code.
SignMuon is a 1-bit, matrix-aware optimizer for distributed training that combines signSGD's majority-vote sign aggregation with Muon's polar-step framework, achieving 32x bandwidth reduction over float32 while maintaining strong convergence and performance on benchmarks like CIFAR-10/ResNet-50 and nanoGPT.