NVIDIA FLARE Optimizes Federated Multimodal AI Training

NVIDIA has detailed new capabilities within its open-source NVIDIA FLARE framework designed to orchestrate federated training for large vision-language models. To demonstrate these capabilities, NVIDIA collaborated with William & Mary to develop FedUMM, a framework for federated unified multimodal models. FedUMM freezes a BLIP backbone and exchanges only lightweight LoRA adapters across distributed clients. In testing with up to 16 clients under Dirichlet-controlled heterogeneity, this adapter-only approach reduced per-client communication from 28.6 GB to just 0.094 GB per round.

Despite the massive reduction in data transmission, the FedUMM approach maintained high accuracy. On the VQA v2 and GenEval benchmarks with eight clients, the model retained approximately 97 percent of the performance of a centralized reference model. Furthermore, the adapter-only federation actually improved VQA v2 scores by 0.7 points compared to a full-model federated averaging baseline. This demonstrates that practitioners do not need to sacrifice model quality to achieve massive bandwidth savings.

For workflows that require full-model updates or larger adapters, NVIDIA FLARE 2.8.0 introduces several engineering solutions to handle heavy payloads. The FLARE Tensor Downloader uses a pull-based protocol to stream PyTorch tensors incrementally, reducing peak memory usage during model distribution. To prevent server CPU memory from scaling linearly with the number of clients, a tensor disk offload module writes incoming PyTorch FedAvg updates to temporary safetensors files, loading them only when needed. Additionally, large-object externalization replaces massive objects with lightweight references.

These developments significantly lower the barrier to entry for institutions training vision-language models on decentralized datasets. By utilizing the NVIDIA FLARE Recipe API, developers can easily define client update contracts and transition from simulated environments to multi-site deployments. Practitioners can now choose between parameter-efficient methods like FedUMM, CreamFL, FedCLIP, and FedPIA, or leverage streaming and disk offloading to run full-model training across resource-constrained nodes.