Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
Summary
This paper introduces Pion, a novel spectrum-preserving optimizer for large language model training that uses orthogonal equivalence transformations to maintain singular values during weight updates, offering stable performance comparable to standard optimizers.
View Cached Full Text
Cached at: 05/13/26, 12:15 PM
Paper page - Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
Source: https://huggingface.co/papers/2605.12492
Abstract
Pion is a spectrum-preserving optimizer for large language model training that uses orthogonal equivalence transformations to maintain singular values during weight updates, offering stable performance comparable to standard optimizers.
We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based onorthogonal equivalence transformation. Unlikeadditive optimizerssuch asAdamandMuon, Pion updates each weight matrix through left and rightorthogonal transformations, preserving itssingular valuesthroughout training. This yields anoptimization mechanismthat modulates the geometry ofweight matriceswhile keeping theirspectral normfixed. We derive the Pion update rule, systematically examine its design choices, and analyze itsconvergence behavioralong with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.
View arXiv pageView PDFProject pageGitHub3Add to collection
Get this paper in your agent:
hf papers read 2605\.12492
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.12492 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.12492 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.12492 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
This paper introduces Pion, a new optimizer that replaces Muon's spectral whitening with a high-pass NS iteration to stabilize training in low-rank and low-SNR regimes, achieving improved performance in VLA and RLVR tasks.
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
This paper identifies a geometric mismatch in the Dion low-rank spectral optimizer and proposes Orth-Dion, which replaces column normalization with QR orthogonalization to close the convergence gap to full-rank methods like Muon at the same communication cost, validated on large-scale language model pre-training.
Anytime Training with Schedule-Free Spectral Optimization
This paper introduces SF-NorMuon, a schedule-free spectral optimizer that matches or exceeds tuned AdamW on language models up to 772M parameters, with theoretical guarantees for stationarity and long-horizon stability.
Muon$^p$: Muon with Fractional Spectral Powers
This paper introduces Muon^p, a novel optimizer that uses fractional spectral-power updates to interpolate between Muon and gradient descent, providing theoretical justification and empirical gains on billion-scale fine-tuning tasks.
Aurora: A Leverage-Aware Spectral Optimizer
Aurora is a leverage-aware spectral optimizer that addresses neuron death in MLP layers by enforcing row uniformity while preserving the polar factor geometry of Muon updates, achieving state-of-the-art performance on the modded-nanoGPT speedrun benchmark.