Paper page - Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

Source: https://huggingface.co/papers/2606.15007 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Nemotron 3 Ultra is a large-scale language model featuring hybrid Mamba-Attention architecture with 550 billion parameters, achieving high inference throughput and extended context length through specialized training techniques.

We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameterMixture-of-ExpertsHybrid Mamba-Attentionlanguage model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained usingSupervised Fine Tuning(SFT),Reinforcement Learning(RL), andMulti-teacher On-Policy Distillation(MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies -LatentMoE,Multi Token Prediction(MTP),NVFP4pre-training, multi-environmentRLVR, MOPD, andreasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2606\.15007

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.15007 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.15007 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.15007 in a Space README.md to link it from this page.

Collections including this paper4