Leverage a PyTorch-native fine-tuning library with Day-0 Hugging Face checkpoint support for the recently released Alibaba open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), its largest open-weight model.

Developers can post-train the model for domain-specific use cases with configurable reasoning using PyTorch-native tools and NVIDIA NeMo AutoModel. Train directly on existing checkpoints without model conversion, with support for full SFT or memory-efficient LoRA fine-tuning.

Read the full post: https://bit.ly/4xdN25l

Serve Qwen3.8-2.4T-A95B, a 2.4T-Parameter Model, with Configurable Reasoning on NVIDIA GB300

Source: https://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/ Alibaba released the open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), its largest open-weight model, bringing near-frontier capabilities to the open ecosystem. It has 2.4T total parameters with 95B activated per token. It’s a fine-grained mixture of experts (MoE) architecture with a hybrid of full and linear attention, a context window of up to one million tokens, and an output length of up to 128K, designed for demanding reasoning and agentic workloads.

Deploying a 2.4T parameter open-weight model requires data-center-scale accelerated compute. Inference at this scale depends on extreme co-design across chips, system architecture, and software. NVIDIA is working with the open-source ecosystem to bring the model to multinode deployments through optimized kernels, inference runtimes, and distributed serving recipes.

Without additional model tuning, the model achieves a throughput of over 4K tokens per second per GPU and over 350 tokens per second per user on NVIDIA GB300 NVL72 in FP8 precision on Day 0. Further optimizations, including NVFP4 precision, are expected to deliver enhanced performance gains over time.

Architectural innovations for long-context inferencehttps://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/#architectural_innovations_for_long-context_inference%C2%A0

Qwen3.8-2.4T-A95B is built for the hardest agentic workloads like coding, large-scale document analysis, and long-running multi-step workflows. Unlike chat-first models that send a single prompt and receive a single reply, agentic applications accumulate system instructions, tool outputs, retrieved documents, code, logs, and multi-step reasoning traces across a workflow. As context grows, attention, compute, and KV cache memory become the binding constraints.

The full-attention and linear-attention hybrid architecture addresses this, and the model alternates between the two. In the full-attention layers, every token attends to every other token, and in the linear-attention layers, the growing KV cache is replaced with a bounded recurrent state. Qwen3.8-2.4T-A95B keeps both compute and memory bounded as context scales to up to one million tokens.

Fine-grained MoE makes the 2.4T parameter count practical to serve. Instead of a small number of large experts, capacity is distributed across a larger population of smaller experts, improving specialization and routing efficiency per unit of activated compute. A learned router activates only the experts needed per token, so serving costs track active parameters, not the full 2.4T parameters, delivering frontier-scale capacity at a fraction of the cost of a comparable dense model.

Built-in reasoning controls (low/high/xhigh) enable developers to configure inference depth per request, trading compute for reasoning quality depending on the task: dial up for complex multi-step reasoning or dial down for high-throughput document processing.

Figure 1. Overview of the Qwen3.8-2.4T-A95B linear gated delta networks plus full attention with fine-grained MoE architecture

Qwen3.8-2.4T-A95B optimized performance on GB300 NVL72https://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/#qwen38-24t-a95b_optimized_performance_on_gb300%E2%80%AFnvl72%C2%A0

The GB300 NVL72 features a rack-scale architecture that integrates 72 NVIDIA Blackwell Ultra GPUs into a single platform. Its large, 72-GPU NVIDIA NVLink domain enables efficient all-to-all communication at 130 TB/s, eliminating bottlenecks that appear when expert traffic must cross traditional off-the-shelf networks.

Out of the box, Qwen3.82.4T-A95B running on NVIDIA Blackwell GB300 NVL72 delivers over4K tokens per second per GPU and over 350 tokens per second per user, enabling AI factories to run large-parameter models in production at high throughput and low latency.

Figure 2. A Pareto curve showing Qwen3.8-2.4T-A95B achieving over 4K tokens per second per GPU at peak throughput on NVIDIA GB300 NVL72

Post-train Qwen3.8-2.4T-A95B and choose a serving pathhttps://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/#post-train_qwen38-24t-a95b_and_choose_a_serving_path

NVIDIA supports multiple inference stacks to meet a variety of developer needs.SGLang,vLLM, andNVIDIA Dynamoprovide open-source inference recipes for developers who require greater control over performance on the NVIDIA-accelerated platform.

It’s also available todeployvia amodel-free NVIDIA NIM, a single inference container that serves any supported model. Download the model weights and deploy on Day-0 to serve fine-tuned checkpoints, and scale to production.

Developers can post-train the model for domain-specific use cases usingNVIDIA NeMo AutoModel, a PyTorch-native fine-tuning library with Day-0 Hugging Face checkpoint support. Train directly on existing checkpoints without model conversion, with support for full SFT or memory-efficient LoRA fine-tuning.

Get started with Qwen3.8-2.4T-A95Bhttps://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/#get_started_with_qwen38-24t-a95b%C2%A0%C2%A0

Try out the model from these hosted APIs:DeepInfra,DigitalOcean,Fireworks AI,Modal, andOpenRouter

Download Qwen3.8-2.4T-A95B model weights fromHugging FaceorModelScopeanddeploy withamodel-free NVIDIA NIMfrom NVIDIA NGC.

About the Authors