PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Editorial record

Plain-language summary

Instead of replicating the full model on every worker as standard data parallelism does, FSDP splits parameters, gradients, and optimizer state into shards held by different workers; before a layer runs it all-gathers that layer's parameters, then frees them afterward, and reduce-scatters gradients during the backward pass. Communication is overlapped with computation and grouped into units to limit memory spikes and latency. Delivered as a native PyTorch API, it let practitioners train models with hundreds of billions of parameters on commodity clusters without a separate framework.