OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
Summary
OmniOpt presents a unified framework for optimizer selection in large-scale model training, combining meta-pipeline transformations, norm-constrained linear minimization oracles, and a cross-domain benchmark to systematically analyze optimizer families and their trade-offs.
View Cached Full Text
Cached at: 07/07/26, 06:42 AM
Paper page - OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
Source: https://huggingface.co/papers/2607.04033 Authors:
,
,
,
,
,
,
,
,
,
,
Abstract
OmniOpt presents a unified framework for optimizer selection in large-scale model training by combining meta-pipeline transformations, norm-constrained linear minimization oracles, and a cross-domain benchmark to systematically analyze optimizer families and their trade-offs across different training objectives and model scales.
Optimizer selectionforlarge-scale model traininghas become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stagemeta-pipeline, and show that most methods engage only one or two of these stages. Second, we usenorm-constrained linear minimization oracles(LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurabletraining objectivesit aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unifiedcross-domain benchmarkspanning representative optimizers,model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out theirtrade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.
View arXiv pageView PDFProject pageGitHub5Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.04033 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.04033 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.04033 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
optimize_anything: A Universal API for Optimizing any Text Parameter
This paper presents optimize_anything, a universal LLM-based optimization system for text artifacts that achieves state-of-the-art results across diverse tasks including agent architecture discovery, scheduling, CUDA kernel generation, and packing, demonstrating general-purpose text optimization.
OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining
This paper introduces OPTScientist, a theory-guided multi-agent framework for automatically discovering typed optimizer programs using a domain-specific language and closed-loop experimentation. The framework discovered RS-MR, a reduced-state matrix optimizer that improves transformer pretraining over strong baselines.
Optimistic Dual Averaging Unifies Modern Optimizers
This paper introduces SODA, a generalization of Optimistic Dual Averaging that unifies various modern optimizers like Muon and Lion. It proposes a practical wrapper that improves performance across different scales without requiring additional hyperparameter tuning for weight decay.
OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding
OmniPro is the first benchmark for evaluating proactive streaming video understanding in omni-modal large language models, featuring 2,700 samples covering diverse tasks and dual-mode evaluation protocols.
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
This paper presents OmniClean, a visually debiased evaluation benchmark for omni-modal language models, and proposes OmniBoost, a three-stage post-training recipe that enables a 3B model to match the performance of a 30B model on the cleaned benchmark.