arxiv.orgpaper

Accelerating Sharded Data Parallelism at Scale with Federated Learning

This paper introduces FL+FSDP and FL+HSDP, hybrid algorithms interleaving sharded data parallelism with FedAvg-style federated aggregation to reduce communication overhead when training foundation models across large, heterogeneous GPU clusters. Decoupling large data-parallel deployments into smaller, loosely-coupled federation groups bounds global batch size growth and cuts inter-group traffic. A Llama3.1 8B pre-training run on 512 A100 GPUs shows up to 8.04x faster data processing and 4.48x lower evaluation perplexity than standard FSDP/HSDP under identical hyperparameters.