PyTorch can be used to build federated multimodal AI workflows, addressing the challenges caused by distributed data.

To streamline these complex pipelines, NVIDIA FLARE externalizes large message payloads by replacing big objects with lightweight references and transferring data independently. Flare Tensor Downloader further optimizes model distribution through incremental tensor streaming to reduce peak memory requirements.

Read the latest technical blog from @nvidia to learn how to coordinate multi-site federated training and manage large model updates using externalization, tensor streaming, and disk-backed aggregation.

Link to blog: https://bit.ly/4y5J6TW

Building Federated Multimodal AI Workflows with NVIDIA FLARE

Source: https://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/ Modernvision-language models (VLMs)can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data needed to adapt these models may be distributed across institutions or organizations that cannot centralize their raw records.

Federated learningprovides a way to coordinate training across these data-local sites. For VLMs, the challenge is not only orchestration. Sites may contribute different task or modality mixes, and model updates can be large enough to strain network bandwidth and server memory.

This post focuses on two design decisions for federated multimodal AI workflows: what model state should cross the network, and how should it be transferred and aggregated efficiently? It shows howNVIDIA FLAREcoordinates federated training across sites and handles large model updates through externalization, tensor streaming, and disk-backed aggregation.

Those design questions also apply to unified multimodal models (UMMs), which support multiple modalities and tasks within a shared architecture.FedUMM, developed through a collaboration between William & Mary and NVIDIA, provides a concrete example by federating lightweight adapters over a frozen multimodal backbone.

FedUMM is supported by theNVIDIA Academic Grant Program, and received an Outstanding Student Paper Award at theFL@FM workshopat TheWebConf 2026.

Why are VLMs difficult to federate?https://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#why_are_vlms_difficult_to_federate

In a centralized vision-language experiment, images, captions, visual question answering examples, and generation prompts can feed one training pipeline. In a federated setting, those examples are distributed across sites with different data, task mixes, and operating constraints.

This creates two engineering problems. First, sites may train on different task or modality mixes, so the workflow must define what each client updates and how those updates are combined. Second, full-model updates can be expensive to serialize, transfer, and hold in server memory.

The first decision is therefore what to federate. Some approaches exchange distilled knowledge rather than model weights. Others freeze a pretrained backbone and aggregate only lightweight trainable components.CreamFLillustrates the first approach, whileFedCLIP,FedPIA, andFedUMMillustrate the second. When larger updates are required, the system must also support streaming and memory-efficient aggregation.

NVIDIA FLARE, an open source, extensible Python SDK and framework for federated learning and collaborative computing**,**can support both parameter-efficient and full-model communication patterns.

Figure 1 shows the general workflow. Sites keep different mixes of images, text, and prompts local, while the server coordinates training and aggregates approved model updates. Large-object externalization, tensor streaming, and disk-backed aggregation help manage larger payloads.

Figure 1. A federated multimodal AI workflow with NVIDIA FLARE

Coordinating training across clientshttps://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#coordinating_training_across_clients_

Every NVIDIA FLARE job separates global coordination from local execution. The server schedules rounds and aggregates updates, while each client trains or evaluates against its local data. Site-specific preprocessing, prompt construction, and batching remain inside the client.

TheNVIDIA FLARE Recipe APIprovides a concise starting point. TheFedAvgrecipe pairs a model with a client training script. The same recipe can be run in simulation or in a real provisioned multi-site deployment.

Before implementing the model, define the client update contract: what remains local, what may leave the site, which model components each client may update, and which metrics return to the server. When clients update different model components, the contract should also specify how those component-level updates are combined.

Moving and aggregating large model updates efficientlyhttps://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#moving_and_aggregating_large_model_updates_efficiently

A common baseline method for federated VLM training is fine-tuning and aggregating the whole model parameters. This will lead to large model updates. With many clients joining the federation, they create two distinct memory pressures: serializing and transferring one update, and holding several client updates in memory during aggregation. NVIDIA FLARE supports several features to tackle this challenge.

Externalize large objectshttps://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#externalize_large_objects

NVIDIA FLARE can replace large objects in a message with lightweight references and transfer the underlying data separately. This keeps the control message small and supports payloads that exceed the ordinary serialized-message limit. Built-in decomposers cover PyTorch tensors, NumPy arrays, and common FLARE structures; custom decomposers are needed only for application-specific object types.

Stream tensorshttps://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#stream_tensors

For PyTorch workflows,FLARE Tensor Downloaderstreams tensors incrementally using a pull-based protocol. Only the requested chunk is serialized at a time, reducing peak memory during model distribution. Chunk size can be tuned to balance request overhead against per-chunk memory. TensorFlow workflows use the traditional serialization path.

Offload aggregation to diskhttps://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#offload_aggregation_to_disk

Streaming reduces memory pressure during transfer, but the server may still need to hold several client updates during aggregation. This causes the server’s peak memory to grow linearly with the number of clients. In NVIDIA FLARE 2.8.0,tensor disk offloadwrites incoming PyTorch FedAvg updates to temporarysafetensorsfiles and loads them as needed, thus preventing the linear growth in CPU memory.

These mechanisms complement payload reduction from adapter-based training to enable full-model training, larger adapters, or federated learning with many clients.

For tested configuration examples, see theNVIDIA FLARE Recipe API,FLARE Tensor Downloader, andtensor disk offload documentation.

Federating lightweight adapters over a frozen VLM with FedUMMhttps://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#federating_lightweight_adapters_over_a_frozen_vlm_with_fedumm

FedUMMprovides a concrete example of minimizing what crosses the network in NVIDIA FLARE. Each simulated client keeps a frozenBLIP backboneand trains LoRA adapters locally. NVIDIA FLARE coordinates the rounds and aggregates only the adapter updates. FedUMM is designed for generality with modality-specific encoders for vision, audio, and text, while its current experiments focus on vision-language.

The reported experiments evaluateVQA v2andGenEvalunder Dirichlet-controlled heterogeneity with up to 16 clients. In an eight-client comparison, adapter-only federation reduced per-client communication from 28.6 GB to 0.094 GB per round and improved VQA v2 by 0.7 points relative to full-model FedAvg. At eight clients, performance remained about 97% of the centralized reference on both benchmarks (Figure 2).

Figure 2. FedUMM trains LoRA adapters over a frozen BLIP3o backbone and aggregates only the trainable components The evaluation uses simulated sites, synthetic partitions, and public general-domain benchmarks. It does not establish clinical performance or formal privacy guarantees; it shows that raw training data remains local within the simulated federated workflow.

FedUMM reduces the system burdens at its source by exchanging only small LoRA adapters. Not every AI workflow can do that. When clients must send larger updates, tensor streaming reduces memory pressure during transfer; when the server must aggregate updates from many clients, disk-backed aggregation reduces how much data must be held in memory.

Checklist for designing federated multimodal AI workflowshttps://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#checklist_for_designing_federated_multimodal_ai_workflows

When designing a federated multimodal workflow, consider both what each client should contribute and how those updates will move through the system. The following checklist summarizes the key decisions for balancing model quality, communication cost, and system memory requirements.

• **Define the update contract:**Decide what stays local, what each client sends, and how the updates are combined.
• **Minimize the payload:**Exchange lightweight adapters when possible, and full-model updates only when the task requires them.
• **Choose how updates move and aggregate:**Use externalization and tensor streaming for large in-memory updates, together with disk offload on server when aggregation several updates would exceed server memory.
• **Evaluate end-to-end:**Measure model quality together with communication, runtime, memory use, data heterogeneity, and failures.

Get started building federated multimodal AI workflowshttps://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare/#get_started_building_federated_multimodal_ai_workflows%C2%A0

Start by defining the update contract for your workflow: what remains local, what model state each client may return, and which clients should contribute to each aggregation. Use theNVIDIA FLARE Recipe APIto implement the workflow and validate it in simulation at the expected client count.

Next, choose the payload-handling mechanism that matches your bottleneck. Usehttps://nvflare.readthedocs.io/en/2.8.0/programming_guide/file_streaming.htmllarge-object externalizationandFLARE Tensor Downloaderfor large model updates, together with tensor disk offload when server-side aggregation memory becomes a constraint.

For a concrete adapter-based example, explore the paper,FedUMM: A General Framework for Federated Learning with Unified Multimodal Models, and itsimplementation in the NVIDIA FLARE repository. After establishing a working baseline, useAuto-FLto adapt and tune the federated experiment for your own datasets and tasks.

To learn more, join us forNVIDIA Flare Day 2026, a free online event that explores cutting-edge applications of federated learning across industries.