nvidia/Nemotron-Labs-Audex-30B-A3B · Hugging Face

Source: https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#introductionIntroduction

We’re excited to introduceNemotron-Labs-Audex-30B-A3B, a unified audio-text LLM built onNemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM with 30B MoE model with 3B activated parameters. Audex-30B-A3B extends the vocabulary for discrete audio tokens used for speech and general audio outputs, as well as an audio encoder for speech and general audio inputs. Audex-30B-A3B delivers strong abilities on audio tasks (audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation) while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. Audex-30B-A3B operates in boththinkingandinstruct(non-thinking) modes.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#model-architectureModel Architecture

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#templatesTemplates

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#multi-stage-sft-and-cascaded-rl-pipelinesMulti-Stage-SFT and Cascaded-RL Pipelines

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#quick-startQuick Start

• Audex-30B-A3B follows the ChatML template and supports both thinking and instruct (non-thinking) modes. Reasoning content is enclosed within<think\>and</think\>tags. To activate the instruct (non-thinking) mode, we prepend<think\></think\>to the beginning of the assistant’s response.
• Audex-30B-A3B supports up to a 1M-token context length.
• Audex-30B-A3B follows Nemotron-Cascade-2 on text evaluation.
• Audex-30B-A3B has different recommended inference setups per audio-related task as described below.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#environmentEnvironment

We usevLLM 0.20.0 container image:vllm/vllm-openai:v0.20.0-cu129

• vLLM inference— text-only reasoning, text-to-speech, text-to-audio, and audio understanding / speech recognition / speech translation: runs onvLLM 0.20.0.
• Hugging Face / transformers inference— requires transformers >= 4.53.0 (tested with 4.53.3) and also works with transformers >= 5.0. This additionally needsmamba\-ssmandcausal\-conv1d(build against your CUDA toolchain, e.g.pip install \-\-no\-build\-isolation causal\-conv1d==1\.6\.2\.post1 mamba\-ssm==2\.3\.2\.post1).

Audio extras:vllm/vllm\-openai:v0\.20\.0image does not include audio codecs. This command installs audio-related packages:python3 \-m pip install "vllm\[audio\]".

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#audio-qa-inferenceAudio QA Inference

Audio QA includes audio understanding, speech recognition, and speech translation (see templates in Introduction).

• vLLM (recommended)— offlineLLM\.generateand an OpenAI-compatibleaudio\_urlserver.
• Hugging Face / transformers— requires transformers >= 4.53.0 (we tested with 4.53.3), and also works with transformers >= 5.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#inputsInputs

To prepare inputs, create a JSON file in the following format with the<sound\>\\nplaceholder:

[
  {
    "id": "sample_0",
    "sound": "/path/to/audio_0.wav",
    "conversations": [
      {"from": "human", "value": "<sound>\nDescribe the audio in detail."},
      {"from": "gpt", "value": "N/A"}
    ]
  },
  {
    "id": "sample_1",
    "sound": "/path/to/audio_1.wav",
    "conversations": [
      {"from": "human", "value": "<sound>\n{prompt}"},
      {"from": "gpt", "value": "N/A"}
    ]
  },
  ...
]

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#inference-recipesInference recipes

• For audio understanding, we use top_p=0.9 and temperature=0.7.
• For speech recognition and translation, we use greedy sampling.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#vllm-recommendedvLLM (recommended)

To install environments:

python3 -m pip install "vllm[audio]"   # audio input decoding; skip if your image already bundles it (see Environment)
pip install -e inference_scripts_vllm/audioqa_scripts --no-deps --no-build-isolation

• Offline (JSON in → JSONL out):

python inference_scripts_vllm/audioqa_scripts/run_audioqa_vllm.py \
  --model-path "$(pwd)/checkpoint_folder_full" \
  --input-json ./inputs.json \
  --output-jsonl ./audioqa_outputs/results.jsonl \
  --tensor-parallel-size 8

• OpenAI-compatible server + client:

bash inference_scripts_vllm/audioqa_scripts/serve_audioqa_vllm.sh "$(pwd)/checkpoint_folder_full" 8000
python inference_scripts_vllm/audioqa_scripts/client_audioqa.py --audio /path/to/audio.wav --prompt "Describe this audio."

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#hugging-face–transformersHugging Face / transformers

• Example inference script:bash inference\_scripts\_hf/inference\_example\.sh.
• Task instruction examples: audio understanding — a question about the audio; speech recognition —Transcribe the speech in the input audio\.\\n<sound\>; speech translation — a translation instruction such asTranslate the speech in the input audio into English\.\\n<sound\>.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#audio-generation-inferenceAudio Generation Inference

Audio generation includes text-to-speech and text-to-audio generation.

First, prepare vLLM inference usingbash model\_conversion\_scripts/prepare\_audiogen\_vllm\_checkpoint\.sh(which only creates symlinks of safetensors undercheckpoint\_folder\_audiogen).

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#text-to-audio-ttaText-to-audio (TTA)

DownloadXCodec1(hf-audio/xcodec-hubert-general-balanced) via

hf download hf-audio/xcodec-hubert-general-balanced --local-dir /path/to/xcodec1

Prepare a folder/path/to/caption\_txt\_dir/with all .txt files where each contains one caption. Runcd inference\_scripts\_vllm/audiogen\_scripts/and set\-\-tensor\-parallel\-sizeto the number of GPUs. Run

XCODEC1_PATH=/path/to/xcodec1 python3 run_audio_gen_vllm_rvq_logit_mask.py \
        --task tta \
        --model-path $(pwd)/../../checkpoint_folder_audiogen/ \
        --dataset-path /path/to/caption_txt_dir/ \
        --output-dir ../../tta_outputs/dataset_name/ \
        --tensor-parallel-size 8 \
        --temperature 1.0 \
        --top-k 80 \
        --max-tokens 2048 \
        --cfg-scale 3.0 \
        --cfg-pairs-per-batch 2

Finally (optional), apply the 48 kHz enhancement VAE to the generated waveforms; seeenhancement\_VAE/README\.md.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#text-to-speech-ttsText-to-speech (TTS)

we recommend using the standalone Audex causal speech decoder inaudex\_causal\_speech\_decoder(default).

./run_tts_vllm.sh --transcription "The weather is so good, and I want to enjoy the beautiful morning in the park." \
    --output-dir ./tts_outputs --utt-id the_weather_is_so_good

Alternatively, users can download the original XCodec2 viathis repoand decode the tokens after full generation; this has better quality but is not streaming.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#text-only-reasoningText-Only Reasoning

The text reasoning followsNemotron-Cascade-2-30B-A3B.

• To create the checkpoint that completely matches their setup, runpython model\_conversion\_scripts/convert\_full\_HF\_to\_textonly\_HF\.pyto remove the audio-related vocabularies.
• Then, the text-only inference completely follows Nemotron-Cascade-2-30B-A3B. See a simple reasoning example withcd inference\_scripts\_vllm/textonly\_scripts/; python run\_text\_vllm\_example\.py \-\-model\-path $\(pwd\)/\.\./\.\./checkpoint\_folder\_textonly.
• Note: you could also avoid model conversion by usingsampling\_params = SamplingParams\(allowed\_token\_ids=list\(range\(131072\)\)\)in vLLM inference to mask the audio tokens, although we did not thoroughly test this approach.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#demo-speech-to-speech-interaction-with-text-reasoningDemo: speech-to-speech interaction with text reasoning

Seeinference\_scripts\_vllm/unified\_s2s\_scripts/README\.md.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#reproducibilityReproducibility

The benchmark numbers below use the following setups:

• **Text-to-speech (TTS):**the original non-streaming XCodec2 decoder.
• **Text-to-audio (TTA):**XCodec1 followed by the enhancement VAE.
• **Text:**same as Nemotron-Cascade-2.
• **Audio understanding:**transformers 4.53.3 and Megatron-LM’s native inference.

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#detailed-benchmark-resultsDetailed Benchmark Results

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#text-resultsText Results

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#text-to-speech-resultsText-To-Speech Results

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#text-to-audio-resultsText-To-Audio Results

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#speech-recognition-and-translation-resultsSpeech Recognition and Translation Results

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#audio-understanding-resultsAudio Understanding Results

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#speech-to-speech-resultsSpeech-To-Speech Results

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#release-dateRelease Date

June 8, 2026

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#licenseLicense

Your use of this model is governed by theNVIDIA Oneway Noncommercial License

https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B#citationCitation

@article{Nemotron-Labs-Audex,
  title={Unified Audio Intelligence Without Regressing on Text Intelligence},
  author={Kong, Zhifeng and Lee, Sang-gil and Kim, Jaehyeon and Wang, Boxin and Liu, Zihan and Kim, Sungwon and Chen, Yang and Goel, Arushi and Roy, Rajarshi and Dai, Wenliang and Yang, Zhuolin and Chen, Yangyi and Jiang, Dongfu and Ghosh, Sreyan and Rintamaki, Tuomas and Tao, Andrew and Raiman, Jonathan and Shoeybi, Mohammad and Catanzaro, Bryan and Ping, Wei},
  year={2026}
}