Paper page - Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

Source: https://huggingface.co/papers/2607.05722 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Nemotron-Labs-Diffusion is a tri-mode language model that combines autoregressive, diffusion, and self-speculation decoding to achieve superior throughput and efficiency compared to existing models.

We introduce Nemotron-Labs-Diffusion, a tri-modelanguage model(LM) that unifies AR, diffusion, andself-speculation decodingwithin a single architecture. Trained with ajoint AR-diffusion objective, Nemotron-Labs-Diffusion can switch modes to sustain highthroughputacross deployment settings and concurrency levels. Our study shows that (1) AR and diffusion objectives are complementary: diffusion improveslookahead planning, while AR provides left-to-rightlinguistic priors. (2) In self-speculation mode, diffusion drafts while AR verifies, outperformingmulti-token prediction(MTP) methods in both acceptance rate and real-device efficiency. (3) Aspeed-of-light analysisfurther demonstrates diffusion’s long-term potential, with up to 76.5% more tokens per forward pass than self-speculation under an optimal sampler. Scaling to 3B, 8B, and 14B parameters, our Nemotron-Labs-Diffusion family, including base, instruct, and vision-language models, consistently outperforms state-of-the-art open-source AR and diffusion LMs in both accuracy and speed. For example, Nemotron-Labs-Diffusion-8B decodes 6x more tokens per forward than Qwen3-8B with comparable accuracy, translating to 4x higherthroughputonSPEED-BenchwithSGLangon aGB200 GPU.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.05722

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.05722 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.05722 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.05722 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.