@RisingSayak: Tensor parallel loading in Diffusers got a massive upgrade On Flux.2-Dev DiT, with a TP degree of 4 (A10G): • 30.4s → 1…

X AI KOLs Following Tools

Summary

Hugging Face Diffusers 上张量并行(tensor parallel)加载迎来重大优化:在 Flux.2-Dev DiT、TP=4(A10G)配置下,加载时间从 30.4s 降至 12.5s(约 2.4 倍提速),每 rank 峰值 CPU 内存从 64.1 GB 降至 6.8 GB(减少约 89%)。相关分布式推理(Accelerate 与 PyTorch Distributed)用法已更新到官方文档。

Tensor parallel loading in Diffusers got a massive upgrade On Flux.2-Dev DiT, with a TP degree of 4 (A10G): • 30.4s → 12.5s loading • 64.1 → 6.8 GB peak CPU memory/rank ~2.4× faster loading. ~89% less CPU memory. Check it out: https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#tensor-parallelism…
Original Article
View Cached Full Text

Cached at: 10/03/26, 07:00 PM

Tensor parallel loading in Diffusers got a massive upgrade On Flux.2-Dev DiT, with a TP degree of 4 (A10G): • 30.4s → 12.5s loading • 64.1 → 6.8 GB peak CPU memory/rank ~2.4× faster loading. ~89% less CPU memory. Check it out: https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#tensor-parallelism…


Distributed inference · Hugging Face

Source: https://huggingface.co/docs/diffusers/main/en/training/distributed_inference Distributed inference splits the workload across multiple GPUs. It a useful technique for fitting larger models in memory and can process multiple prompts for higher throughput.

This guide will show you how to useAccelerateandPyTorch Distributedfor distributed inference.

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#accelerateAccelerate

Accelerate is a library designed to simplify inference and training on multiple accelerators by handling the setup, allowing users to focus on their PyTorch code.

Install Accelerate with the following command.

uv pip install accelerate

Initialize aaccelerate.PartialStateclass in a Python file to create a distributed environment. Theaccelerate.PartialStateclass manages process management, device control and distribution, and process coordination.

Move theDiffusionPipelinetoaccelerate\.PartialState\.deviceto assign a GPU to each process.

import torch
from accelerate import PartialState
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    "Qwen/Qwen-Image", dtype=torch.float16
)
distributed_state = PartialState()
pipeline.to(distributed_state.device)

Use thesplit_between_processesutility as a context manager to automatically distribute the prompts between the number of processes.

with distributed_state.split_between_processes(["a dog", "a cat"]) as prompt:
    result = pipeline(prompt).images[0]
    result.save(f"result_{distributed_state.process_index}.png")

Callaccelerate launchto run the script and use the\-\-num\_processesargument to set the number of GPUs to use.

accelerate launch run_distributed.py --num_processes=2

Refer to this minimal examplescriptfor running inference across multiple GPUs. To learn more, take a look at theDistributed Inference with 🤗 Accelerateguide.

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#pytorch-distributedPyTorch Distributed

PyTorchDistributedDataParallelenablesdata parallelism, which replicates the same model on each device, to process different batches of data in parallel.

Importtorch\.distributedandtorch\.multiprocessinginto a Python file to set up the distributed process group and to spawn the processes for inference on each GPU.

import torch
import torch.distributed as dist
import torch.multiprocessing as mp

from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    "Qwen/Qwen-Image", dtype=torch.float16,
)

Create a function for inference withinit_process_group. This method creates a distributed environment with the backend type, therankof the current process, and theworld\_sizeor number of processes participating (for example, 2 GPUs would beworld\_size=2).

Move the pipeline torankand useget\_rankto assign a GPU to each process. Each process handles a different prompt.

def run_inference(rank, world_size):
    dist.init_process_group("nccl", rank=rank, world_size=world_size)

    pipeline.to(rank)

    if torch.distributed.get_rank() == 0:
        prompt = "a dog"
    elif torch.distributed.get_rank() == 1:
        prompt = "a cat"

    image = sd(prompt).images[0]
    image.save(f"./{'_'.join(prompt)}.png")

Usemp.spawnto create the number of processes defined inworld\_size.

def main():
    world_size = 2
    mp.spawn(run_inference, args=(world_size,), nprocs=world_size, join=True)

if __name__ == "__main__":
    main()

Calltorchrunto run the inference script and use the\-\-nproc\_per\_nodeargument to set the number of GPUs to use.

torchrun --nproc_per_node=2 run_distributed.py

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#devicemapdevice_map

Thedevice\_mapargument enables distributed inference by automatically placing model components on separate GPUs. This is especially useful when a model doesn’t fit on a single GPU. You can usedevice\_mapto selectively load and unload the required model components at a given stage as shown in the example below (assumes two GPUs are available).

Setdevice\_map="balanced"to evenly distributes the text encoders on all available GPUs. You can use themax\_memoryargument to allocate a maximum amount of memory for each text encoder. Don’t load any other pipeline components to avoid memory usage.

from diffusers import FluxPipeline
import torch

prompt = """
cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
"""

pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    transformer=None,
    vae=None,
    device_map="balanced",
    max_memory={0: "16GB", 1: "16GB"},
    dtype=torch.bfloat16
)
with torch.no_grad():
    print("Encoding prompts.")
    prompt_embeds, pooled_prompt_embeds, text_ids = pipeline.encode_prompt(
        prompt=prompt, prompt_2=None, max_sequence_length=512
    )

After the text embeddings are computed, remove them from the GPU to make space for the diffusion transformer.

import gc 

def flush():
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.reset_max_memory_allocated()
    torch.cuda.reset_peak_memory_stats()

del pipeline.text_encoder
del pipeline.text_encoder_2
del pipeline.tokenizer
del pipeline.tokenizer_2
del pipeline

flush()

Setdevice\_map="auto"to automatically distribute the model on the two GPUs. This strategy places a model on the fastest device first before placing a model on a slower device like a CPU or hard drive if needed. The trade-off of storing model parameters on slower devices is slower inference latency.

from diffusers import AutoModel
import torch 

transformer = AutoModel.from_pretrained(
    "black-forest-labs/FLUX.1-dev", 
    subfolder="transformer",
    device_map="auto",
    dtype=torch.bfloat16
)

Runpipeline\.hf\_device\_mapto see how the various models are distributed across devices. This is useful for tracking model device placement. You can also callhf\_device\_mapon the transformer model to see how it is distributed.

Add the transformer model to the pipeline and set theoutput\_type="latent"to generate the latents.

pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    text_encoder=None,
    text_encoder_2=None,
    tokenizer=None,
    tokenizer_2=None,
    vae=None,
    transformer=transformer,
    dtype=torch.bfloat16
)

print("Running denoising.")
height, width = 768, 1360
latents = pipeline(
    prompt_embeds=prompt_embeds,
    pooled_prompt_embeds=pooled_prompt_embeds,
    num_inference_steps=50,
    guidance_scale=3.5,
    height=height,
    width=width,
    output_type="latent",
).images

Remove the pipeline and transformer from memory and load a VAE to decode the latents. The VAE is typically small enough to be loaded on a single device.

import torch
from diffusers import AutoencoderKL
from diffusers.image_processor import VaeImageProcessor

vae = AutoencoderKL.from_pretrained(ckpt_id, subfolder="vae", dtype=torch.bfloat16).to("cuda")  
vae_scale_factor = 2 ** (len(vae.config.block_out_channels) - 1)
image_processor = VaeImageProcessor(vae_scale_factor=vae_scale_factor)

with torch.no_grad():
    print("Running decoding.")
    latents = FluxPipeline._unpack_latents(latents, height, width, vae_scale_factor)
    latents = (latents / vae.config.scaling_factor) + vae.config.shift_factor

    image = vae.decode(latents, return_dict=False)[0]
    image = image_processor.postprocess(image, output_type="pil")
    image[0].save("split_transformer.png")

By selectively loading and unloading the models you need at a given stage and sharding the largest models across multiple GPUs, it is possible to run inference with large models on consumer GPUs.

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#context-parallelismContext parallelism

Context parallelismsplits input sequences across multiple GPUs to reduce memory usage. Each GPU processes its own slice of the sequence.

Useset_attention_backend()to switch to a more optimized attention backend. Refer to thistablefor a complete list of available backends.

Most attention backends are compatible with context parallelism. Open anissueif a backend is not compatible.

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ring-attentionRing Attention

Key (K) and value (V) representations communicate between devices usingRing Attention. This ensures each split sees every other token’s K/V. Each GPU computes attention for its local K/V and passes it to the next GPU in the ring. No single GPU holds the full sequence, which reduces communication latency.

Pass aContextParallelConfigto theparallel\_configargument of the transformer model. The config supports thering\_degreeargument that determines how many devices to use for Ring Attention.

import torch
from torch import distributed as dist
from diffusers import DiffusionPipeline, ContextParallelConfig

def setup_distributed():
    if not dist.is_initialized():
        dist.init_process_group(backend="nccl")
    rank = dist.get_rank()
    device = torch.device(f"cuda:{rank}")
    torch.cuda.set_device(device)
    return device

def main():
    device = setup_distributed()
    world_size = dist.get_world_size()

    pipeline = DiffusionPipeline.from_pretrained(
        "black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16
    ).to(device)
    pipeline.transformer.set_attention_backend("_native_cudnn")

    cp_config = ContextParallelConfig(ring_degree=world_size)
    pipeline.transformer.enable_parallelism(config=cp_config)

    prompt = """
    cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
    highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
    """

    
    generator = torch.Generator().manual_seed(42)
    image = pipeline(
        prompt,
        guidance_scale=3.5,
        num_inference_steps=50,
        generator=generator,
    ).images[0]

    if dist.get_rank() == 0:
        image.save(f"output.png")

    if dist.is_initialized():
        dist.destroy_process_group()

if __name__ == "__main__":
    main()

The script above needs to be run with a distributed launcher, such astorchrun, that is compatible with PyTorch.\-\-nproc\-per\-nodeis set to the number of GPUs available.

torchrun --nproc-per-node 2 above_script.py

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ulysses-attentionUlysses Attention

Ulysses Attentionsplits a sequence across GPUs and performs anall-to-allcommunication (every device sends/receives data to every other device). Each GPU ends up with all tokens for only a subset of attention heads. Each GPU computes attention locally on all tokens for its head, then performs another all-to-all to regroup results by tokens for the next layer.

ContextParallelConfigsupports Ulysses Attention through theulysses\_degreeargument. This determines how many devices to use for Ulysses Attention.

Pass theContextParallelConfigtoenable\_parallelism\(\).

pipeline.transformer.enable_parallelism(config=ContextParallelConfig(ulysses_degree=2))

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#unified-attentionUnified Attention

Unified Sequence Parallelismcombines Ring Attention and Ulysses Attention into a single approach for efficient long-sequence processing. It applies Ulysses’sall-to-allcommunication first to redistribute heads and sequence tokens, then uses Ring Attention to process the redistributed data, and finally reverses theall-to-allto restore the original layout.

This hybrid approach leverages the strengths of both methods:

  • Ulysses Attentionefficiently parallelizes across attention heads
  • Ring Attentionhandles very long sequences with minimal memory overhead
  • Together, they enable 2D parallelization across both heads and sequence dimensions

ContextParallelConfigsupports Unified Attention by specifying bothulysses\_degreeandring\_degree. The total number of devices used isulysses\_degree \* ring\_degree, arranged in a 2D grid where Ulysses and Ring groups are orthogonal (non-overlapping). Pass theContextParallelConfigwith bothulysses\_degreeandring\_degreeset to bigger than 1 toenable\_parallelism\(\).

pipeline.transformer.enable_parallelism(config=ContextParallelConfig(ulysses_degree=2, ring_degree=2))

Unified Attention is to be used when there are enough devices to arrange in a 2D grid (at least 4 devices).

We ran a benchmark with Ulysess, Ring, and Unified Attention withthis scripton a node of 4 H100 GPUs. The results are summarized as follows:

CP BackendTime / Iter (ms)Steps / SecPeak Memory (GB)ulysses6670.7897.5033.85ring13076.4923.8256.02unified_balanced11068.7054.5233.85From the above table, it’s clear that Ulysses provides better throughput, but the number of devices it can use remains limited to the number of attention heads, a limitation that is solved by unified attention.

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ulysses-anything-attentionUlysses Anything Attention

The default Ulysses Attention mechanism requires that the sequence length of hidden states must be divisible by the number of devices. This imposes significant limitations on the practical application of Ulysses Attention.Ulysses Anything Attentionis a variant of Ulysses Attention that supports arbitrary sequence lengths and arbitrary numbers of attention heads, thereby enhancing the versatility of Ulysses Attention in practical use.

ContextParallelConfigsupports Ulysses Anything Attention by specifying bothulysses\_degreeandulysses\_anything. Please note that Ulysses Anything Attention is not currently supported by Unified Attention. Pass theContextParallelConfigwith bothulysses\_degreeset to bigger than 1 andulysses\_anything=Truetoenable\_parallelism\(\).

pipeline.transformer.enable_parallelism(config=ContextParallelConfig(ulysses_degree=2, ulysses_anything=True))

To avoid multiple forced CUDA sync caused by H2D and D2H transfers, please add thegloobackend ininit\_process\_group. This will significantly reduce communication latency.

We ran a benchmark for FLUX.1-dev with Ulysses, Ring, Unified Attention and Ulysses Anything Attention withthis scripton a node of 4 L20 GPUs. The results are summarized as follows:

CP BackendTime / Iter (ms)Steps / SecPeak Memory (GB)Shape (HxW)ulysses281.073.5637.111024x1024ring351.342.8537.011024x1024unified_balanced324.373.0837.161024x1024ulysses_anything280.943.5637.111024x1024ulyssesfailedfailedfailed1008x1008ringfailedfailedfailed1008x1008unified_balancedfailedfailedfailed1008x1008ulysses_anything278.403.5936.991008x1008From the above table, it is clear that Ulysses Anything Attention offers better compatibility with arbitrary sequence lengths while maintaining the same performance as the standard Ulysses Attention.

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ring-anything-attentionRing Anything Attention

The defaultRing Attentionrequires the sequence length of hidden states to be evenly divisible across the ring degree.Ring Anything Attentionis a variant of Ring Attention that supports arbitrary (non-evenly divisible) sequence lengths. It pads each rank’s local KV to the global maximum sequence length, all-gathers the padded KV buffer, and slices back to each rank’s true length before running attention.

Ring Anything Attention is not supported by Unified Attention. Setring\_degree \> 1andring\_anything=Trueto enable Ring Anything Attention.

pipeline.transformer.enable_parallelism(config=ContextParallelConfig(ring_degree=2, ring_anything=True))

Add thegloobackend toinit_process_groupto avoid multiple forced CUDA syncs from H2D and D2H transfers.

import torch.distributed as dist

dist.init_process_group(backend="cpu:gloo,cuda:nccl")

Ring Anything Attention only currently supports inference and non-Noneattention masks aren’t supported.attn\_maskmust beNone.

See the FLUX.1-dev benchmarks below on a node of 4 RTX 4090 (48GB) GPUs.

CP BackendTime / Iter (ms)Steps / SecPeak Memory (GB)Shape (HxW)ulysses259.073.8633.831024x1024ring338.982.9533.831024x1024unified_balanced321.543.1133.831024x1024ulysses_anything259.073.8633.831024x1024ring_anything340.142.9433.831024x1024ulyssesfailedfailedfailed1008x1008ringfailedfailedfailed1008x1008unified_balancedfailedfailedfailed1008x1008ulysses_anything253.163.9533.751008x1008ring_anything335.572.9833.751008x1008From the above table, Ring Anything Attention offers compatibility with arbitrary sequence lengths while maintaining performance comparable to the standard Ring Attention. For more details on the motivation and trade-offs for Ring Anything Attention, seethis comment.

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#parallelconfigparallel_config

Passparallel\_configduring model initialization to enable context parallelism.

CKPT_ID = "black-forest-labs/FLUX.1-dev"

cp_config = ContextParallelConfig(ring_degree=2)
transformer = AutoModel.from_pretrained(
    CKPT_ID, 
    subfolder="transformer", 
    dtype=torch.bfloat16,
    parallel_config=cp_config
)

pipeline = DiffusionPipeline.from_pretrained(
    CKPT_ID, transformer=transformer, dtype=torch.bfloat16,
).to(device)

https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#tensor-parallelismTensor parallelism

Tensor parallelismshards the weight matrices of a model across devices. Each device holds a column-wise ("colwise") or row-wise ("rowwise") slice of each layer, computes a partial result, and anAllReduce/AllGatherat the layer boundary reconstructs the full output. Unlike context parallelism, it reduces the per-deviceweightmemory, which is useful for models that do not fit on a single device.

Pass aTensorParallelConfigto theparallel\_configargument of the model’sfrom_pretrained().tp\_degreeis the number of devices to shard across and must divide the model’s number of attention heads. The model must define a\_tp\_plan(a flat mapping of module-name globs to a"colwise"/"rowwise"style).

Loading this way shards the checkpointwhile reading it: each rank reads only its own slice of each sharded weight and places it straight onto its own device. Nothing full-size is ever materialized, so per-rank memory falls astp\_degreerises.

Compared to loading the full model and then callingenable\_parallelism\(\), it loads faster and uses less CPU memory per rank, with the gap growing astp\_degreerises. Numbers below are forblack-forest-labs/FLUX.2-dev(transformer only, 32B params, bf16) on 4x A10G (23GB).

tp_degreemethodload timepeak CPU/rank4from\_pretrained\(parallel\_config=\.\.\.\)12.5s6.8GB4from\_pretrained+enable\_parallelism30.4s64.1GB``` import torch from torch import distributed as dist from diffusers import DiffusionPipeline, Flux2Transformer2DModel, TensorParallelConfig

def main(): dist.init_process_group(backend=“nccl”) rank, world_size = dist.get_rank(), dist.get_world_size() device = torch.device(f“cuda:{rank}“) torch.cuda.set_device(device)

transformer = Flux2Transformer2DModel.from_pretrained(
    "black-forest-labs/FLUX.2-dev",
    subfolder="transformer",
    torch_dtype=torch.bfloat16,
    parallel_config=TensorParallelConfig(tp_degree=world_size),
)

pipeline = DiffusionPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-dev", transformer=transformer, torch_dtype=torch.bfloat16
)


pipeline.text_encoder.to(device)
pipeline.vae.to(device)

generator = torch.Generator().manual_seed(42)
image = pipeline(prompt="a cat holding a sign that says hello", generator=generator).images[0]
if rank == 0:
    image.save("output.png")
dist.destroy_process_group()

if name == “main”: main()


torchrun –nproc-per-node 4 tensor_parallel_flux.py


`tp\_degree`is taken from`world\_size`above, so`\-\-nproc\-per\-node 4`shards the transformer across 4 devices\.

> Loading with a tensor\-parallel`parallel\_config`isn’t supported yet with`device\_map`,`quantization\_config`,`low\_cpu\_mem\_usage=False`,`use\_flashpack=True`, or non\-safetensors weights; each raises rather than quietly falling back to loading the full checkpoint\. Combining tensor parallelism with quantization, offloading, or LoRA adapters isn’t supported yet either, so those raise however the model is sharded\. To shard a model that is already in memory, call`enable\_parallelism\(\)`with the same config instead — that loads everything first and reshards it, so it costs full checkpoint memory on every rank\.

Saving a tensor\-parallel model isn’t supported yet, and[save\_pretrained\(\)](https://huggingface.co/docs/diffusers/main/en/api/models/overview#diffusers.ModelMixin.save_pretrained)raises on one\. Save the model before sharding it\.

### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#writing-a-tensor-parallelism-plan](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#writing-a-tensor-parallelism-plan)Writing a tensor parallelism plan

Tensor parallelism only works on models that define a`\_tp\_plan`, a flat class attribute mapping module\-name globs to a sharding style\. Writing one is mostly a matter of pairing each projection that*expands*the hidden dimension with the projection that*contracts*it back\.

Each key may contain**at most one`\*`**, and the prefix before it must resolve to an[`nn\.ModuleList`](https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html)so a single entry covers every block\. A key without a`\*`applies to the model itself\. Paths are relative to the model\.

#### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#colwise-and-rowwise](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#colwise-and-rowwise)Colwise and rowwise

StyleShardsEach rankUse for`"colwise"`output features \(`weight`dim 0\)computes a slice of the output`to\_q`,`to\_k`,`to\_v`, FFN in\-projection`"rowwise"`input features \(`weight`dim 1\)computes a partial sum`to\_out\.0`, FFN out\-projectionAlways pair them in that order\. A`"colwise"`projection leaves its output sharded, the following`"rowwise"`projection consumes that shard directly, and a single`AllReduce`at the block boundary reconstructs the result\. Sharding the pair any other way forces a gather in the middle and communicates far more\.

For attention this means each rank owns a subset of heads, which is why`tp\_degree`must divide the head count\. Encoder\-stream duplicates \(`add\_q\_proj`,`to\_add\_out`,`ff\_context`\) follow the same pattern as their image\-stream counterparts\.

#### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#fused-projections](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#fused-projections)Fused projections

When one`Linear`packs several logical tensors along the dimension being sharded, plain`"colwise"`/`"rowwise"`slices straight across the concatenation and misaligns the pieces\. Use`PackedColwiseParallel`/`PackedRowwiseParallel`instead, which shard each packed block independently\.

from …hooks.tensor_parallel import PackedColwiseParallel, PackedRowwiseParallel


`blocks`is a list of proportional integers whose sum divides the packed dimension —`\[1, 1\]`for a SwiGLU gate\+up projection of equal halves, or`\[1, 1, 1, 3, 3\]`for a fused Q\+K\+V\+gate\+up projection with`mlp\_ratio=3`\.

“transformer_blocks.*.ff.linear_in”: PackedColwiseParallel([1, 1]),


When the block sizes are only known from the config, omit the argument and store the absolute sizes on the`Linear`during`\_\_init\_\_`instead, as`\_tp\_packed\_col\_blocks`or`\_tp\_packed\_row\_blocks`\.

self.to_out._tp_packed_row_blocks = [self.inner_dim, self.mlp_hidden_dim]


“single_transformer_blocks.*.attn.to_out”: PackedRowwiseParallel(),


Both forms appear in one plan in[`transformer\_flux2\.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux2.py): the double\-stream SwiGLU pair passes`blocks`literally, while the fused single\-stream QKV\+MLP projection stores its sizes on the`Linear`in`Flux2ParallelSelfAttention\.\_\_init\_\_`because they depend on`mlp\_ratio`and`mlp\_mult\_factor`\.

#### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#what-to-leave-out](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#what-to-leave-out)What to leave out

Anything absent from the plan stays replicated on every rank, which is the right choice for normalization layers, AdaLN modulation \(`img\_mod`/`txt\_mod`\), patch and text embeddings, and the final`norm\_out`/`proj\_out`\. These are small, so sharding them saves little memory while adding communication\.

#### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#constraints-and-verification](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#constraints-and-verification)Constraints and verification

- `tp\_degree`must divide`config\.num\_attention\_heads`\.
- Every packed block must*individually*be divisible by`tp\_degree`, not just their sum\.

Both are validated by[from\_pretrained\(\)](https://huggingface.co/docs/diffusers/main/en/api/models/overview#diffusers.ModelMixin.from_pretrained)and`enable\_parallelism\(\)`before any weight is loaded or sharded\.

Validate a new plan numerically rather than by eye: generate with a fixed seed on a single device, then again under tensor parallelism, and compare the outputs\. A misplaced`"colwise"`/`"rowwise"`usually still runs and produces a plausible but wrong image\.

> Start from an existing plan for a similar architecture\.[QwenImageTransformer2DModel](https://huggingface.co/docs/diffusers/main/en/api/models/qwenimage_transformer2d#diffusers.QwenImageTransformer2DModel)is fully unfused and every entry is plain`"colwise"`/`"rowwise"`,[FluxTransformer2DModel](https://huggingface.co/docs/diffusers/main/en/api/models/flux_transformer#diffusers.FluxTransformer2DModel)adds a single packed row\-wise projection, and[Flux2Transformer2DModel](https://huggingface.co/docs/diffusers/main/en/api/models/flux2_transformer#diffusers.Flux2Transformer2DModel)covers both packed styles — see[`transformer\_qwenimage\.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_qwenimage.py),[`transformer\_flux\.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py), and[`transformer\_flux2\.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux2.py)respectively\.

## [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#choosing-a-strategy](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#choosing-a-strategy)Choosing a strategy

The strategies above solve different problems, and the useful question is not which is fastest in the abstract but what you are running out of\.

StrategySplitsReducesLatency for one promptBest when[Accelerate](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#accelerate)/[DDP](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#pytorch-distributed)prompts across replicasnothing — each device holds a full copyunchangedthe model already fits and you have many prompts[`device\_map`](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#devicemap)components across devicesweight memoryslightly worsethe model doesn’t fit and the interconnect is slow[Context parallelism](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#context-parallelism)the input sequenceactivation memorylowersequences are long — high resolution or video[Tensor parallelism](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#tensor-parallelism)weight matricesweight memorylowerone component’s weights don’t fit and the interconnect is fastSome practical guidance:

- **Throughput on many prompts, model already fits\.**Use data parallelism\. It is the only strategy here that scales throughput linearly without touching the model, and it leaves single\-prompt latency alone\.
- **A single component’s weights don’t fit\.**Reach for tensor parallelism first, since it lowers both memory and latency\. It communicates at every block boundary, so it wants a fast interconnect like NVLink; over PCIe that per\-layer traffic can outweigh the compute it saves, and`device\_map`becomes the better choice\.`device\_map`also handles the case where the components are individually fine but collectively too large\.
- **Activations, not weights, are the problem\.**This is the long\-sequence regime — large images, many frames — and context parallelism is the direct answer\. Pick a backend from the[Ulysses/Ring benchmarks](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ulysses-attention)above\.
- **Both weights and sequence are too large\.**Combine tensor and context parallelism\.[TensorParallelConfig](https://huggingface.co/docs/diffusers/main/en/api/parallel#diffusers.TensorParallelConfig)accepts a`mesh`argument so both can share one device mesh\. This combination is not yet validated, so treat it as experimental\.

[Updateon GitHub](https://github.com/huggingface/diffusers/blob/main/docs/source/en/training/distributed_inference.md)

Similar Articles

@reprompting: reading about the NVIDIA hopper architecture today https://arxiv.org/pdf/2501.12084

X AI KOLs Timeline

这篇论文对 NVIDIA Hopper GPU 架构进行了多层级微基准测试分析,评估了 L2 分区缓存、第四代张量核心(FP8)、DPX 指令、分布式共享内存(DSM)和张量内存加速器(TMA)等新特性的性能表现,结果显示 TMA 异步编程可实现 1.5 倍矩阵乘法加速,FP8 性能接近 FP16 的两倍,DPX 指令可加速生物信息学算法至少 4.75 倍。