Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel

Source: https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel Back to Articles

• NeMo AutoModel: Same API, More Performance
• Performance Comparison- Nemotron 3 Ultra 550B A55B (full fine-tune, multi-node) - Single-node 30B MoE benchmarks - Qwen3-30B-A3B - Nemotron 3 Nano 30B A3B - Where the speedup comes from
• Transformers v5 Features Leveraged by HuggingFace AutoModel- Expert Backends
• Expert Parallelism and DeepEP- Dynamic Weight Loading
• Getting Started
• Conclusion
• Acknowledgements

HuggingFace Transformers has become the foundation of the open-source AI ecosystem, and the recentTransformers v5release strengthened it with first-class support for Mixture-of-Experts (MoE) models, now the dominant architecture forfrontier models. v5 ships the MoE foundations: expert backends, dynamic weight loading, and distributed execution that make MoE extensible and easy to build on.

NVIDIA NeMo AutoModelis an open library part of theNVIDIA NeMo frameworkfor building custom generative AI models at scale. NeMo AutoModel builds cleanly on top of v5, adding Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels, and it leans on v5’s dynamic weight loading to bring those optimizations to a broad and growing set of model families. The payoff is3.4-3.7x higher training throughputand29-32% less GPU memoryon fine-tuning MoE models than native Transformers v5, using the same from_pretrained() API: a single import line, with no other code changes.

This blog details how this combination works and how users can fine-tune MoE models faster without changing their APIs.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#backgroundBackground

The rise of MoE models has introduced new challenges to efficient training: Routing tokens across hundreds of experts, fusing expert matmuls into a single kernel, sharding weights across GPUs, and overlapping communication with computation all require infrastructure beyond what a general-purpose library provides out of the box.

Transformers v5(“v5”) introduced first-class MoE support such asexpert backends,dynamic weight loading, and tensor parallel plans for distributed execution. In addition, v5 made distributed training first-class by integrating PyTorch’s DeviceMesh directly into from_pretrained().

NeMo AutoModelbuilds on top of v5 by subclassing AutoModelForCausalLM, and adding Expert Parallelism (EP), DeepEP fused all-to-all dispatch, and TransformerEngine kernels. DeepEP is the piece v5 doesn’t have yet: it overlaps communication with expert compute. And because NeMo AutoModel rides v5’s reversible weight conversion to load each model, it can focus its engineering on these reusable core ops instead of per-model checkpoint plumbing, while save_pretrained() still emits standard HF checkpoints that tools like vLLM and SGLang can load.

The next section walks through how the two work together and the performance gains we measured, from full fine-tuningNVIDIA Nemotron 3 Ultra 550B A55Bacross 16 nodes down to single-node models such as Qwen3-30B-A3B andNemotron 3 Nano 30B A3B.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#nemo-automodel-same-api-more-performanceNeMo AutoModel: Same API, More Performance

One of NeMo AutoModel’s goals is API compatibility with HuggingFace Transformers to enable open-source community. NeMoAutoModelForCausalLM subclasses AutoModelForCausalLM, so any code that works with HF models works with AutoModel too.

Here’s what loading a model looks like in both. Only the import changes:

That single import does a lot of work. For popular MoE architectures like Qwen3,NVIDIA Nemotron, GPT-OSS, and DeepSeek V3, NeMo AutoModel shipshand-tuned implementationswith TransformerEngine attention, fused linear layers, and custom expert kernels. For everything else, it falls back to vanilla HF while still applying optimizations likeLiger kernelpatching, among others. And whichever path it takes, the resulting model is ready to scale: pass a device_mesh and you have multi-GPU training without further rewrites.

Where NeMo AutoModel really shines is scaling MoE models to multi-GPU training. To trainNemotron 3 Nano 30B A3Bwith Expert Parallelism across 8 GPUs, one adds the distributed mesh configuration:

import os
import torch
import torch.distributed as dist
from nemo_automodel import NeMoAutoModelForCausalLM
from nemo_automodel.recipes._dist_utils import create_distributed_setup_from_config

dist.init_process_group(backend="nccl")
torch.manual_seed(0)
torch.cuda.set_device(int(os.environ.get("LOCAL_RANK", 0)))

dist_setup = create_distributed_setup_from_config(
    {
        "strategy": "fsdp2",
        "ep_size": 8,
    },
)

model = NeMoAutoModelForCausalLM.from_pretrained(
    "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
    dtype=torch.bfloat16,
    distributed_setup=dist_setup,
)

dist.destroy_process_group()

This gives speed, scalability and memory-optimizations with FSDP2, Expert Parallelism, TransformerEngine kernels and DeepEP dispatch, all from a from_pretrained() call.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#performance-comparisonPerformance Comparison

We evaluated NeMo AutoModel in two regimes: full fine-tuning a frontier-scale 550B model across 16 nodes, and training two 30B MoE models on a single node. The 550B result shows why Expert Parallelism is essential at scale; the 30B results quantify the per-GPU speedup over Transformers v5.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#nemotron-3-ultra-550b-a55b-full-fine-tune-multi-nodeNemotron 3 Ultra 550B A55B (full fine-tune, multi-node)

Nemotron 3 Ultra 550B A55Bis a 550B-parameter hybrid model shipping with Mamba2, LatentMoE, and Multi-Token Prediction (MTP). We benchmark afull fine-tune: every parameter is updated and the Adam optimizer state is materialized, which at this scale spans16 H100 nodes (128 GPUs).

Methodology:

ParameterValueHardware16x H100 80GB (128 GPUs)Expert ParallelismEP=64Local batch size2Sequence length4,096FeaturesMTP, activation checkpointing, fused linear cross-entropyKernelsDeepEP dispatch + torch_mm experts + TransformerEngine MetricNeMo AutoModel (EP=64)TPS/GPU (avg)815TFLOP/s/GPU~293Peak Memory58.2 GiB **Why there is no Transformers v5 column.**Transformers v5 runs out of memory at this scale, so there is no v5 number to report here. AutoModel’s Expert Parallelism shards the experts across GPUs to bring the footprint within budget, which is what lets the full fine-tune run. The 30B comparisons below show the same advantage where v5 fits.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#single-node-30b-moe-benchmarksSingle-node 30B MoE benchmarks

We benchmarked three approaches on a single node with 8x H100 80GB GPUs: HF Transformers v4 (hub code), HF Transformers v5 (with best available optimizations), and NeMo AutoModel (EP=8 + custom kernels).

Methodology:

ParameterValueHardware8x H100 80GB (single node)Sequence length4,096Local batch size1 **A note on the routing gate.**The NeMo AutoModel numbers below use a balanced routing gate, which forces tokens to be distributed uniformly across experts. This emulates theidealoperating point an MoE is trained toward: a well-trained model’s load-balancing loss drives expert utilization to near-uniform, so balanced routing reflects the steady-state a real workload converges to (and removes the straggler noise that random dummy tokens otherwise inject into expert parallelism). v4/v5 run their native router on the same dummy tokens. The balanced gate therefore measures NeMo AutoModel at its target MoE operating point, and the v4/v5 columns reflect their out-of-the-box behavior.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#qwen3-30b-a3bQwen3-30B-A3B

Metricv4v5 (FA2 + grouped_mm)NeMo AutoModel (EP=8)v5 → NeMo AutoModelTPS/GPU (avg)deadlock3,07511,3403.69xPeak Memory—68.2 GiB48.1 GiB**-29%**Avg Forward+Loss—582 ms194 ms3.00xAvg Backward—758 ms178 ms4.26x **Why v4 deadlocks:**Transformers v4 stores Qwen3 MoE experts as a ModuleList of 128 individual MLP modules, each separately FSDP-wrapped. The forward pass uses a data-dependent loop that only iterates experts that received tokens. With different data per rank, different ranks skip different experts, causing mismatched FSDP AllGather/ReduceScatter collectives and an indefinite hang. Transformers v5 fixes this by storing experts as fused 3D parameter tensors (no per-expert modules, no per-expert FSDP collectives).

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#nemotron-3-nano-30b-a3bNemotron 3 Nano 30B A3B

Metricv4 (hub code)v5 (FA2 + grouped_mm + Mamba CUDA)NeMo AutoModel (EP=8)v5 → NeMo AutoModelTPS/GPU (avg)1,8074,58315,4213.36xPeak Memory61.9 GiB62.1 GiB42.5 GiB**-32%**Avg Forward+Loss1,024 ms283 ms109 ms2.60xAvg Backward1,246 ms611 ms157 ms3.89x **v4 config:**trust_remote_code=True (NVIDIA’s hub modeling code). The hub code’s expert loop is FSDP-safe (iterates all experts regardless of token assignment), so it doesn’t deadlock like Qwen3 v4.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#where-the-speedup-comes-fromWhere the speedup comes from

The 3.4-3.7x speedup from NeMo AutoModel over Transformers v5 comes from three sources:

1. **Expert Parallelism reduces memory pressure.**EP=8 distributes expert weights across GPUs, cutting the per-GPU MoE footprint by 8x. For Qwen3, this drops peak memory from 68.2 GiB to 48.1 GiB (-29%). For Nemotron Nano, it drops from 62.1 GiB to 42.5 GiB (-32%), freeing headroom for larger batch sizes or longer sequences.
2. **DeepEP fuses communication with computation.**Instead of separate AllGather/ReduceScatter collectives for expert routing, DeepEP fuses token dispatch and combines into optimized GPU kernels, overlapping communication with expert computation.
3. **TransformerEngine kernels accelerate core operations.**TE’s fused attention, linear layers, and RMSNorm implementations provide consistent speedups over their PyTorch/Flash Attention equivalents across all layer types, not just MoE layers.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#transformers-v5-features-leveraged-by-huggingface-automodelTransformers v5 Features Leveraged by HuggingFace AutoModel

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#expert-backendsExpert Backends

One of the most impactful features in Transformers v5 is theexperts_implementationparameter, which includes three expert backends:

BackendDescriptionBest foreagerFor-loop over selected expertsDebugging, compatibility, and correctness. Also available for v4.batched_mmDuplicates expert params, single batched GEMM via torch.bmmSmall inputs, fast with torch.compile. Added for v5grouped_mmOrders tokens by expert, single grouped GEMM via torch.nn.functional.grouped_mmTraining (memory efficient, no param duplication). Added for v5. The grouped_mm backend is the key training optimization: instead of looping over experts one by one, it sorts tokens by their assigned expert and executes a single fused grouped matrix multiplication.

NeMo AutoModel takes this further. For models with custom implementations, it uses DeepEP fused all-to-all dispatch combined with grouped GEMM kernels and TransformerEngine linear layers. The progression looks like:

v4 (eager for-loop) → v5 (grouped_mm) → NeMo AutoModel (DeepEP + GMM + TE)

In NeMo AutoModel, the expert backend is configured through BackendConfig:

from nemo_automodel.components.models.common.utils import BackendConfig

backend = BackendConfig(
    attn="te",           # TransformerEngine attention
    linear="te",         # TransformerEngine linear layers
    experts="torch_mm",  # Grouped expert matmul
    dispatcher="deepep", # DeepEP fused all-to-all
)

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#expert-parallelism-and-deepepExpert Parallelism and DeepEP

Transformers v5 also ships anExpert Parallelism path. It shards expert weights across GPUs. TheGroupedGemmParallelstyle loads only each device’s local experts, andRouterParallelroutes tokens and combines results with an all_reduce. It’s neatly built on v5’s existing tensor-parallel machinery. Enabling it makes the model’s tp_plan return itsexpert plan, so expert parallelism shares the device budget with data parallelism (ep × dp = world_size). For the single-node 30B benchmarks here, we found plain data-parallel v5 (dp=8, ep=1) to be the fastest v5 configuration, so that’s the v5 setup we report.

NeMo AutoModel takes a complementary approach tuned for multi-GPU MoE training. It makes EP its own parallelism dimension, a dedicated moe_mesh alongside (rather than carved from) the data-parallel mesh, using PyTorch’s DTensor with Shard(0). Because the expert mesh is orthogonal to data parallelism, the two compose on the same devices. On 8 GPUs NeMo AutoModel runs ep=8 and dp=8 together, so every GPU trains on its own data shard while holding only 1/8 of the experts. Expert weights are physically sharded across GPUs along the expert dimension.

# From nemo_automodel/components/moe/parallelizer.py
from torch.distributed.tensor import Shard, distribute_tensor

# Each GPU holds only 1/ep_size of the expert weights
distribute_tensor(param, device_mesh, [Shard(0)])

With ep_size=8 on 8 GPUs, each GPU holds only 1/8 of the expert parameters. For a model like Nemotron-3-Nano-30B-A3B with ~55 GiB of expert weights, EP reduces the per-GPU expert footprint from ~55 GiB to ~6.8 GiB, making training possible where FSDP-only approaches run out of memory.

On top of EP, NeMo AutoModel integratesDeepEPthat fuses the token routing into optimized GPU kernels, and delivers significant speedups when combined with grouped GEMM for grouped expert computation. In ourlarge-scale MoE benchmarks, DeepEP + grouped GEMM reduced cost per iteration by 47% on the full DeepSeek V3 671B model compared to all-gather + looped expert baselines.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#dynamic-weight-loadingDynamic Weight Loading

Transformers v5 also introduced adynamic weight loadingsystem through WeightConverter and WeightRenaming. This enables MoE checkpoint to be stored in fused 3D tensors for more efficient execution. The WeightConverter applies composable operations to transform checkpoint tensors on-the-fly during from_pretrained().

NeMo AutoModel is a direct consumer of this v5 API. Over20 model typesuse this mechanism through MODELS_REQUIRING_TENSOR_MERGING, including Mixtral, Qwen2 MoE, Qwen3 MoE, DeepSeek V2/V3, OLMoE, and more. The conversions are fully reversible: save_pretrained() produces standard HF-format checkpoints that any downstream tool can load.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#getting-startedGetting Started

To try NeMo AutoModel, please visit our official documentation page toget started.

For more details, see:

• NeMo AutoModel HuggingFace API Compatibility Guide
• NeMo AutoModel Model Coverage
• NeMo AutoModel Performance Summary
• NeMo AutoModel on HuggingFace

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#conclusionConclusion

NVIDIA NeMo AutoModel is the natural next step for HuggingFace users scaling up model training. By building directly on Transformers v5, AutoModel provides a zero-friction upgrade path: change one import line and get a model instance that is more than three times as fast.

On Qwen3-30B-A3B and Nemotron 3 Nano 30B-A3B, this delivers 3.4-3.7x higher training throughput with 29-32% less GPU memory compared to the best Transformers v5 configuration. And because true Expert Parallelism shards experts across GPUs, the same path scales up to full fine-tuning a 550B model like Nemotron 3 Ultra across 16 nodes, the regime where Expert Parallelism becomes essential to fit the model in memory. Because NeMo AutoModel checkpoints are standard HF-format safetensors, you can deploy them on inference frameworks like vLLM and SGLang.

The code, configs, and benchmark scripts are all available in theNeMo AutoModel repository.

https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel#acknowledgementsAcknowledgements

Core contributors to this work, listed alphabetically by last name: Adil Asif, Hemil Desai, Alexandros Koumparoulis, and Huiying Li.