@RisingSayak: Tensor parallel loading in Diffusers got a massive upgrade On Flux.2-Dev DiT, with a TP degree of 4 (A10G): • 30.4s → 1…
Summary
Hugging Face Diffusers 上张量并行(tensor parallel)加载迎来重大优化:在 Flux.2-Dev DiT、TP=4(A10G)配置下,加载时间从 30.4s 降至 12.5s(约 2.4 倍提速),每 rank 峰值 CPU 内存从 64.1 GB 降至 6.8 GB(减少约 89%)。相关分布式推理(Accelerate 与 PyTorch Distributed)用法已更新到官方文档。
View Cached Full Text
Cached at: 10/03/26, 07:00 PM
Tensor parallel loading in Diffusers got a massive upgrade On Flux.2-Dev DiT, with a TP degree of 4 (A10G): • 30.4s → 12.5s loading • 64.1 → 6.8 GB peak CPU memory/rank ~2.4× faster loading. ~89% less CPU memory. Check it out: https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#tensor-parallelism…
Distributed inference · Hugging Face
Source: https://huggingface.co/docs/diffusers/main/en/training/distributed_inference Distributed inference splits the workload across multiple GPUs. It a useful technique for fitting larger models in memory and can process multiple prompts for higher throughput.
This guide will show you how to useAccelerateandPyTorch Distributedfor distributed inference.
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#accelerateAccelerate
Accelerate is a library designed to simplify inference and training on multiple accelerators by handling the setup, allowing users to focus on their PyTorch code.
Install Accelerate with the following command.
uv pip install accelerate
Initialize aaccelerate.PartialStateclass in a Python file to create a distributed environment. Theaccelerate.PartialStateclass manages process management, device control and distribution, and process coordination.
Move theDiffusionPipelinetoaccelerate\.PartialState\.deviceto assign a GPU to each process.
import torch
from accelerate import PartialState
from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained(
"Qwen/Qwen-Image", dtype=torch.float16
)
distributed_state = PartialState()
pipeline.to(distributed_state.device)
Use thesplit_between_processesutility as a context manager to automatically distribute the prompts between the number of processes.
with distributed_state.split_between_processes(["a dog", "a cat"]) as prompt:
result = pipeline(prompt).images[0]
result.save(f"result_{distributed_state.process_index}.png")
Callaccelerate launchto run the script and use the\-\-num\_processesargument to set the number of GPUs to use.
accelerate launch run_distributed.py --num_processes=2
Refer to this minimal examplescriptfor running inference across multiple GPUs. To learn more, take a look at theDistributed Inference with 🤗 Accelerateguide.
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#pytorch-distributedPyTorch Distributed
PyTorchDistributedDataParallelenablesdata parallelism, which replicates the same model on each device, to process different batches of data in parallel.
Importtorch\.distributedandtorch\.multiprocessinginto a Python file to set up the distributed process group and to spawn the processes for inference on each GPU.
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained(
"Qwen/Qwen-Image", dtype=torch.float16,
)
Create a function for inference withinit_process_group. This method creates a distributed environment with the backend type, therankof the current process, and theworld\_sizeor number of processes participating (for example, 2 GPUs would beworld\_size=2).
Move the pipeline torankand useget\_rankto assign a GPU to each process. Each process handles a different prompt.
def run_inference(rank, world_size):
dist.init_process_group("nccl", rank=rank, world_size=world_size)
pipeline.to(rank)
if torch.distributed.get_rank() == 0:
prompt = "a dog"
elif torch.distributed.get_rank() == 1:
prompt = "a cat"
image = sd(prompt).images[0]
image.save(f"./{'_'.join(prompt)}.png")
Usemp.spawnto create the number of processes defined inworld\_size.
def main():
world_size = 2
mp.spawn(run_inference, args=(world_size,), nprocs=world_size, join=True)
if __name__ == "__main__":
main()
Calltorchrunto run the inference script and use the\-\-nproc\_per\_nodeargument to set the number of GPUs to use.
torchrun --nproc_per_node=2 run_distributed.py
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#devicemapdevice_map
Thedevice\_mapargument enables distributed inference by automatically placing model components on separate GPUs. This is especially useful when a model doesn’t fit on a single GPU. You can usedevice\_mapto selectively load and unload the required model components at a given stage as shown in the example below (assumes two GPUs are available).
Setdevice\_map="balanced"to evenly distributes the text encoders on all available GPUs. You can use themax\_memoryargument to allocate a maximum amount of memory for each text encoder. Don’t load any other pipeline components to avoid memory usage.
from diffusers import FluxPipeline
import torch
prompt = """
cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
"""
pipeline = FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
transformer=None,
vae=None,
device_map="balanced",
max_memory={0: "16GB", 1: "16GB"},
dtype=torch.bfloat16
)
with torch.no_grad():
print("Encoding prompts.")
prompt_embeds, pooled_prompt_embeds, text_ids = pipeline.encode_prompt(
prompt=prompt, prompt_2=None, max_sequence_length=512
)
After the text embeddings are computed, remove them from the GPU to make space for the diffusion transformer.
import gc
def flush():
gc.collect()
torch.cuda.empty_cache()
torch.cuda.reset_max_memory_allocated()
torch.cuda.reset_peak_memory_stats()
del pipeline.text_encoder
del pipeline.text_encoder_2
del pipeline.tokenizer
del pipeline.tokenizer_2
del pipeline
flush()
Setdevice\_map="auto"to automatically distribute the model on the two GPUs. This strategy places a model on the fastest device first before placing a model on a slower device like a CPU or hard drive if needed. The trade-off of storing model parameters on slower devices is slower inference latency.
from diffusers import AutoModel
import torch
transformer = AutoModel.from_pretrained(
"black-forest-labs/FLUX.1-dev",
subfolder="transformer",
device_map="auto",
dtype=torch.bfloat16
)
Run
pipeline\.hf\_device\_mapto see how the various models are distributed across devices. This is useful for tracking model device placement. You can also callhf\_device\_mapon the transformer model to see how it is distributed.
Add the transformer model to the pipeline and set theoutput\_type="latent"to generate the latents.
pipeline = FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
text_encoder=None,
text_encoder_2=None,
tokenizer=None,
tokenizer_2=None,
vae=None,
transformer=transformer,
dtype=torch.bfloat16
)
print("Running denoising.")
height, width = 768, 1360
latents = pipeline(
prompt_embeds=prompt_embeds,
pooled_prompt_embeds=pooled_prompt_embeds,
num_inference_steps=50,
guidance_scale=3.5,
height=height,
width=width,
output_type="latent",
).images
Remove the pipeline and transformer from memory and load a VAE to decode the latents. The VAE is typically small enough to be loaded on a single device.
import torch
from diffusers import AutoencoderKL
from diffusers.image_processor import VaeImageProcessor
vae = AutoencoderKL.from_pretrained(ckpt_id, subfolder="vae", dtype=torch.bfloat16).to("cuda")
vae_scale_factor = 2 ** (len(vae.config.block_out_channels) - 1)
image_processor = VaeImageProcessor(vae_scale_factor=vae_scale_factor)
with torch.no_grad():
print("Running decoding.")
latents = FluxPipeline._unpack_latents(latents, height, width, vae_scale_factor)
latents = (latents / vae.config.scaling_factor) + vae.config.shift_factor
image = vae.decode(latents, return_dict=False)[0]
image = image_processor.postprocess(image, output_type="pil")
image[0].save("split_transformer.png")
By selectively loading and unloading the models you need at a given stage and sharding the largest models across multiple GPUs, it is possible to run inference with large models on consumer GPUs.
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#context-parallelismContext parallelism
Context parallelismsplits input sequences across multiple GPUs to reduce memory usage. Each GPU processes its own slice of the sequence.
Useset_attention_backend()to switch to a more optimized attention backend. Refer to thistablefor a complete list of available backends.
Most attention backends are compatible with context parallelism. Open anissueif a backend is not compatible.
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ring-attentionRing Attention
Key (K) and value (V) representations communicate between devices usingRing Attention. This ensures each split sees every other token’s K/V. Each GPU computes attention for its local K/V and passes it to the next GPU in the ring. No single GPU holds the full sequence, which reduces communication latency.
Pass aContextParallelConfigto theparallel\_configargument of the transformer model. The config supports thering\_degreeargument that determines how many devices to use for Ring Attention.
import torch
from torch import distributed as dist
from diffusers import DiffusionPipeline, ContextParallelConfig
def setup_distributed():
if not dist.is_initialized():
dist.init_process_group(backend="nccl")
rank = dist.get_rank()
device = torch.device(f"cuda:{rank}")
torch.cuda.set_device(device)
return device
def main():
device = setup_distributed()
world_size = dist.get_world_size()
pipeline = DiffusionPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16
).to(device)
pipeline.transformer.set_attention_backend("_native_cudnn")
cp_config = ContextParallelConfig(ring_degree=world_size)
pipeline.transformer.enable_parallelism(config=cp_config)
prompt = """
cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
"""
generator = torch.Generator().manual_seed(42)
image = pipeline(
prompt,
guidance_scale=3.5,
num_inference_steps=50,
generator=generator,
).images[0]
if dist.get_rank() == 0:
image.save(f"output.png")
if dist.is_initialized():
dist.destroy_process_group()
if __name__ == "__main__":
main()
The script above needs to be run with a distributed launcher, such astorchrun, that is compatible with PyTorch.\-\-nproc\-per\-nodeis set to the number of GPUs available.
torchrun --nproc-per-node 2 above_script.py
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ulysses-attentionUlysses Attention
Ulysses Attentionsplits a sequence across GPUs and performs anall-to-allcommunication (every device sends/receives data to every other device). Each GPU ends up with all tokens for only a subset of attention heads. Each GPU computes attention locally on all tokens for its head, then performs another all-to-all to regroup results by tokens for the next layer.
ContextParallelConfigsupports Ulysses Attention through theulysses\_degreeargument. This determines how many devices to use for Ulysses Attention.
Pass theContextParallelConfigtoenable\_parallelism\(\).
pipeline.transformer.enable_parallelism(config=ContextParallelConfig(ulysses_degree=2))
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#unified-attentionUnified Attention
Unified Sequence Parallelismcombines Ring Attention and Ulysses Attention into a single approach for efficient long-sequence processing. It applies Ulysses’sall-to-allcommunication first to redistribute heads and sequence tokens, then uses Ring Attention to process the redistributed data, and finally reverses theall-to-allto restore the original layout.
This hybrid approach leverages the strengths of both methods:
- Ulysses Attentionefficiently parallelizes across attention heads
- Ring Attentionhandles very long sequences with minimal memory overhead
- Together, they enable 2D parallelization across both heads and sequence dimensions
ContextParallelConfigsupports Unified Attention by specifying bothulysses\_degreeandring\_degree. The total number of devices used isulysses\_degree \* ring\_degree, arranged in a 2D grid where Ulysses and Ring groups are orthogonal (non-overlapping). Pass theContextParallelConfigwith bothulysses\_degreeandring\_degreeset to bigger than 1 toenable\_parallelism\(\).
pipeline.transformer.enable_parallelism(config=ContextParallelConfig(ulysses_degree=2, ring_degree=2))
Unified Attention is to be used when there are enough devices to arrange in a 2D grid (at least 4 devices).
We ran a benchmark with Ulysess, Ring, and Unified Attention withthis scripton a node of 4 H100 GPUs. The results are summarized as follows:
CP BackendTime / Iter (ms)Steps / SecPeak Memory (GB)ulysses6670.7897.5033.85ring13076.4923.8256.02unified_balanced11068.7054.5233.85From the above table, it’s clear that Ulysses provides better throughput, but the number of devices it can use remains limited to the number of attention heads, a limitation that is solved by unified attention.
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ulysses-anything-attentionUlysses Anything Attention
The default Ulysses Attention mechanism requires that the sequence length of hidden states must be divisible by the number of devices. This imposes significant limitations on the practical application of Ulysses Attention.Ulysses Anything Attentionis a variant of Ulysses Attention that supports arbitrary sequence lengths and arbitrary numbers of attention heads, thereby enhancing the versatility of Ulysses Attention in practical use.
ContextParallelConfigsupports Ulysses Anything Attention by specifying bothulysses\_degreeandulysses\_anything. Please note that Ulysses Anything Attention is not currently supported by Unified Attention. Pass theContextParallelConfigwith bothulysses\_degreeset to bigger than 1 andulysses\_anything=Truetoenable\_parallelism\(\).
pipeline.transformer.enable_parallelism(config=ContextParallelConfig(ulysses_degree=2, ulysses_anything=True))
To avoid multiple forced CUDA sync caused by H2D and D2H transfers, please add thegloobackend in
init\_process\_group. This will significantly reduce communication latency.
We ran a benchmark for FLUX.1-dev with Ulysses, Ring, Unified Attention and Ulysses Anything Attention withthis scripton a node of 4 L20 GPUs. The results are summarized as follows:
CP BackendTime / Iter (ms)Steps / SecPeak Memory (GB)Shape (HxW)ulysses281.073.5637.111024x1024ring351.342.8537.011024x1024unified_balanced324.373.0837.161024x1024ulysses_anything280.943.5637.111024x1024ulyssesfailedfailedfailed1008x1008ringfailedfailedfailed1008x1008unified_balancedfailedfailedfailed1008x1008ulysses_anything278.403.5936.991008x1008From the above table, it is clear that Ulysses Anything Attention offers better compatibility with arbitrary sequence lengths while maintaining the same performance as the standard Ulysses Attention.
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ring-anything-attentionRing Anything Attention
The defaultRing Attentionrequires the sequence length of hidden states to be evenly divisible across the ring degree.Ring Anything Attentionis a variant of Ring Attention that supports arbitrary (non-evenly divisible) sequence lengths. It pads each rank’s local KV to the global maximum sequence length, all-gathers the padded KV buffer, and slices back to each rank’s true length before running attention.
Ring Anything Attention is not supported by Unified Attention. Setring\_degree \> 1andring\_anything=Trueto enable Ring Anything Attention.
pipeline.transformer.enable_parallelism(config=ContextParallelConfig(ring_degree=2, ring_anything=True))
Add the
gloobackend toinit_process_groupto avoid multiple forced CUDA syncs from H2D and D2H transfers.
import torch.distributed as dist
dist.init_process_group(backend="cpu:gloo,cuda:nccl")
Ring Anything Attention only currently supports inference and non-
Noneattention masks aren’t supported.attn\_maskmust beNone.
See the FLUX.1-dev benchmarks below on a node of 4 RTX 4090 (48GB) GPUs.
CP BackendTime / Iter (ms)Steps / SecPeak Memory (GB)Shape (HxW)ulysses259.073.8633.831024x1024ring338.982.9533.831024x1024unified_balanced321.543.1133.831024x1024ulysses_anything259.073.8633.831024x1024ring_anything340.142.9433.831024x1024ulyssesfailedfailedfailed1008x1008ringfailedfailedfailed1008x1008unified_balancedfailedfailedfailed1008x1008ulysses_anything253.163.9533.751008x1008ring_anything335.572.9833.751008x1008From the above table, Ring Anything Attention offers compatibility with arbitrary sequence lengths while maintaining performance comparable to the standard Ring Attention. For more details on the motivation and trade-offs for Ring Anything Attention, seethis comment.
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#parallelconfigparallel_config
Passparallel\_configduring model initialization to enable context parallelism.
CKPT_ID = "black-forest-labs/FLUX.1-dev"
cp_config = ContextParallelConfig(ring_degree=2)
transformer = AutoModel.from_pretrained(
CKPT_ID,
subfolder="transformer",
dtype=torch.bfloat16,
parallel_config=cp_config
)
pipeline = DiffusionPipeline.from_pretrained(
CKPT_ID, transformer=transformer, dtype=torch.bfloat16,
).to(device)
https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#tensor-parallelismTensor parallelism
Tensor parallelismshards the weight matrices of a model across devices. Each device holds a column-wise ("colwise") or row-wise ("rowwise") slice of each layer, computes a partial result, and anAllReduce/AllGatherat the layer boundary reconstructs the full output. Unlike context parallelism, it reduces the per-deviceweightmemory, which is useful for models that do not fit on a single device.
Pass aTensorParallelConfigto theparallel\_configargument of the model’sfrom_pretrained().tp\_degreeis the number of devices to shard across and must divide the model’s number of attention heads. The model must define a\_tp\_plan(a flat mapping of module-name globs to a"colwise"/"rowwise"style).
Loading this way shards the checkpointwhile reading it: each rank reads only its own slice of each sharded weight and places it straight onto its own device. Nothing full-size is ever materialized, so per-rank memory falls astp\_degreerises.
Compared to loading the full model and then callingenable\_parallelism\(\), it loads faster and uses less CPU memory per rank, with the gap growing astp\_degreerises. Numbers below are forblack-forest-labs/FLUX.2-dev(transformer only, 32B params, bf16) on 4x A10G (23GB).
tp_degreemethodload timepeak CPU/rank4from\_pretrained\(parallel\_config=\.\.\.\)12.5s6.8GB4from\_pretrained+enable\_parallelism30.4s64.1GB```
import torch
from torch import distributed as dist
from diffusers import DiffusionPipeline, Flux2Transformer2DModel, TensorParallelConfig
def main(): dist.init_process_group(backend=“nccl”) rank, world_size = dist.get_rank(), dist.get_world_size() device = torch.device(f“cuda:{rank}“) torch.cuda.set_device(device)
transformer = Flux2Transformer2DModel.from_pretrained(
"black-forest-labs/FLUX.2-dev",
subfolder="transformer",
torch_dtype=torch.bfloat16,
parallel_config=TensorParallelConfig(tp_degree=world_size),
)
pipeline = DiffusionPipeline.from_pretrained(
"black-forest-labs/FLUX.2-dev", transformer=transformer, torch_dtype=torch.bfloat16
)
pipeline.text_encoder.to(device)
pipeline.vae.to(device)
generator = torch.Generator().manual_seed(42)
image = pipeline(prompt="a cat holding a sign that says hello", generator=generator).images[0]
if rank == 0:
image.save("output.png")
dist.destroy_process_group()
if name == “main”: main()
torchrun –nproc-per-node 4 tensor_parallel_flux.py
`tp\_degree`is taken from`world\_size`above, so`\-\-nproc\-per\-node 4`shards the transformer across 4 devices\.
> Loading with a tensor\-parallel`parallel\_config`isn’t supported yet with`device\_map`,`quantization\_config`,`low\_cpu\_mem\_usage=False`,`use\_flashpack=True`, or non\-safetensors weights; each raises rather than quietly falling back to loading the full checkpoint\. Combining tensor parallelism with quantization, offloading, or LoRA adapters isn’t supported yet either, so those raise however the model is sharded\. To shard a model that is already in memory, call`enable\_parallelism\(\)`with the same config instead — that loads everything first and reshards it, so it costs full checkpoint memory on every rank\.
Saving a tensor\-parallel model isn’t supported yet, and[save\_pretrained\(\)](https://huggingface.co/docs/diffusers/main/en/api/models/overview#diffusers.ModelMixin.save_pretrained)raises on one\. Save the model before sharding it\.
### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#writing-a-tensor-parallelism-plan](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#writing-a-tensor-parallelism-plan)Writing a tensor parallelism plan
Tensor parallelism only works on models that define a`\_tp\_plan`, a flat class attribute mapping module\-name globs to a sharding style\. Writing one is mostly a matter of pairing each projection that*expands*the hidden dimension with the projection that*contracts*it back\.
Each key may contain**at most one`\*`**, and the prefix before it must resolve to an[`nn\.ModuleList`](https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html)so a single entry covers every block\. A key without a`\*`applies to the model itself\. Paths are relative to the model\.
#### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#colwise-and-rowwise](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#colwise-and-rowwise)Colwise and rowwise
StyleShardsEach rankUse for`"colwise"`output features \(`weight`dim 0\)computes a slice of the output`to\_q`,`to\_k`,`to\_v`, FFN in\-projection`"rowwise"`input features \(`weight`dim 1\)computes a partial sum`to\_out\.0`, FFN out\-projectionAlways pair them in that order\. A`"colwise"`projection leaves its output sharded, the following`"rowwise"`projection consumes that shard directly, and a single`AllReduce`at the block boundary reconstructs the result\. Sharding the pair any other way forces a gather in the middle and communicates far more\.
For attention this means each rank owns a subset of heads, which is why`tp\_degree`must divide the head count\. Encoder\-stream duplicates \(`add\_q\_proj`,`to\_add\_out`,`ff\_context`\) follow the same pattern as their image\-stream counterparts\.
#### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#fused-projections](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#fused-projections)Fused projections
When one`Linear`packs several logical tensors along the dimension being sharded, plain`"colwise"`/`"rowwise"`slices straight across the concatenation and misaligns the pieces\. Use`PackedColwiseParallel`/`PackedRowwiseParallel`instead, which shard each packed block independently\.
from …hooks.tensor_parallel import PackedColwiseParallel, PackedRowwiseParallel
`blocks`is a list of proportional integers whose sum divides the packed dimension —`\[1, 1\]`for a SwiGLU gate\+up projection of equal halves, or`\[1, 1, 1, 3, 3\]`for a fused Q\+K\+V\+gate\+up projection with`mlp\_ratio=3`\.
“transformer_blocks.*.ff.linear_in”: PackedColwiseParallel([1, 1]),
When the block sizes are only known from the config, omit the argument and store the absolute sizes on the`Linear`during`\_\_init\_\_`instead, as`\_tp\_packed\_col\_blocks`or`\_tp\_packed\_row\_blocks`\.
self.to_out._tp_packed_row_blocks = [self.inner_dim, self.mlp_hidden_dim]
“single_transformer_blocks.*.attn.to_out”: PackedRowwiseParallel(),
Both forms appear in one plan in[`transformer\_flux2\.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux2.py): the double\-stream SwiGLU pair passes`blocks`literally, while the fused single\-stream QKV\+MLP projection stores its sizes on the`Linear`in`Flux2ParallelSelfAttention\.\_\_init\_\_`because they depend on`mlp\_ratio`and`mlp\_mult\_factor`\.
#### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#what-to-leave-out](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#what-to-leave-out)What to leave out
Anything absent from the plan stays replicated on every rank, which is the right choice for normalization layers, AdaLN modulation \(`img\_mod`/`txt\_mod`\), patch and text embeddings, and the final`norm\_out`/`proj\_out`\. These are small, so sharding them saves little memory while adding communication\.
#### [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#constraints-and-verification](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#constraints-and-verification)Constraints and verification
- `tp\_degree`must divide`config\.num\_attention\_heads`\.
- Every packed block must*individually*be divisible by`tp\_degree`, not just their sum\.
Both are validated by[from\_pretrained\(\)](https://huggingface.co/docs/diffusers/main/en/api/models/overview#diffusers.ModelMixin.from_pretrained)and`enable\_parallelism\(\)`before any weight is loaded or sharded\.
Validate a new plan numerically rather than by eye: generate with a fixed seed on a single device, then again under tensor parallelism, and compare the outputs\. A misplaced`"colwise"`/`"rowwise"`usually still runs and produces a plausible but wrong image\.
> Start from an existing plan for a similar architecture\.[QwenImageTransformer2DModel](https://huggingface.co/docs/diffusers/main/en/api/models/qwenimage_transformer2d#diffusers.QwenImageTransformer2DModel)is fully unfused and every entry is plain`"colwise"`/`"rowwise"`,[FluxTransformer2DModel](https://huggingface.co/docs/diffusers/main/en/api/models/flux_transformer#diffusers.FluxTransformer2DModel)adds a single packed row\-wise projection, and[Flux2Transformer2DModel](https://huggingface.co/docs/diffusers/main/en/api/models/flux2_transformer#diffusers.Flux2Transformer2DModel)covers both packed styles — see[`transformer\_qwenimage\.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_qwenimage.py),[`transformer\_flux\.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py), and[`transformer\_flux2\.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux2.py)respectively\.
## [https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#choosing-a-strategy](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#choosing-a-strategy)Choosing a strategy
The strategies above solve different problems, and the useful question is not which is fastest in the abstract but what you are running out of\.
StrategySplitsReducesLatency for one promptBest when[Accelerate](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#accelerate)/[DDP](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#pytorch-distributed)prompts across replicasnothing — each device holds a full copyunchangedthe model already fits and you have many prompts[`device\_map`](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#devicemap)components across devicesweight memoryslightly worsethe model doesn’t fit and the interconnect is slow[Context parallelism](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#context-parallelism)the input sequenceactivation memorylowersequences are long — high resolution or video[Tensor parallelism](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#tensor-parallelism)weight matricesweight memorylowerone component’s weights don’t fit and the interconnect is fastSome practical guidance:
- **Throughput on many prompts, model already fits\.**Use data parallelism\. It is the only strategy here that scales throughput linearly without touching the model, and it leaves single\-prompt latency alone\.
- **A single component’s weights don’t fit\.**Reach for tensor parallelism first, since it lowers both memory and latency\. It communicates at every block boundary, so it wants a fast interconnect like NVLink; over PCIe that per\-layer traffic can outweigh the compute it saves, and`device\_map`becomes the better choice\.`device\_map`also handles the case where the components are individually fine but collectively too large\.
- **Activations, not weights, are the problem\.**This is the long\-sequence regime — large images, many frames — and context parallelism is the direct answer\. Pick a backend from the[Ulysses/Ring benchmarks](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#ulysses-attention)above\.
- **Both weights and sequence are too large\.**Combine tensor and context parallelism\.[TensorParallelConfig](https://huggingface.co/docs/diffusers/main/en/api/parallel#diffusers.TensorParallelConfig)accepts a`mesh`argument so both can share one device mesh\. This combination is not yet validated, so treat it as experimental\.
[Updateon GitHub](https://github.com/huggingface/diffusers/blob/main/docs/source/en/training/distributed_inference.md)
Similar Articles
The space is moving quick. 2x 4060 Tensor Parallel main weights. RPC offload the drafters to amd strix. Got +2-3GB Vram for context at no perf cost.
作者通过修补 llama.cpp,在 2x RTX 4060 张量并行基础上把 DFlash2 草稿模型经 RPC 卸载到 AMD Strix 设备,实现混合 Nvidia+AMD 硬件下上下文从 150k 提升到 195k 且不损失性能,并附上了可复现的构建补丁与 PR 合并步骤。
TorchNeuron: running HF models on AWS Trainium is easier than ever
TorchNeuron 作为 PyTorch 的原生 Trainium 后端发布公测,Hugging Face 的 Transformers、Diffusers 等库可直接将其作为普通设备使用,显著提升了模型支持率并取代 Optimum Neuron 的复杂封装。
@PyTorch: TorchTitan has released major performance and architecture updates in native PyTorch, adding full 5D parallelism suppor…
TorchTitan 发布重大性能与架构更新,新增对 DeepSeek-V4、Qwen3.5、Kimi K3、Muse Spark Open 等 SOTA 模型的完整 5D 并行支持,并推出统一后训练与智能体工作负载的 TitanRL 框架,FlexShard 与 torch_checkpointing 大幅提升大规模训练效率。
TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates
TPHC 将张量并行分片与超连接残差流结合,使各流获得更独立的更新,在不增加参数和FLOPs的情况下以较低训练开销提升了语言模型损失与零样本推理性能。
@reprompting: reading about the NVIDIA hopper architecture today https://arxiv.org/pdf/2501.12084
这篇论文对 NVIDIA Hopper GPU 架构进行了多层级微基准测试分析,评估了 L2 分区缓存、第四代张量核心(FP8)、DPX 指令、分布式共享内存(DSM)和张量内存加速器(TMA)等新特性的性能表现,结果显示 TMA 异步编程可实现 1.5 倍矩阵乘法加速,FP8 性能接近 FP16 的两倍,DPX 指令可加速生物信息学算法至少 4.75 倍。