ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Editorial record

Plain-language summary

The authors observed that standard data parallelism stores a full copy of optimizer states, gradients, and weights on every GPU, wasting memory as models grow. ZeRO shards these three components across the data-parallel group in three progressive stages, reconstructing pieces via communication only when needed, so per-device memory falls roughly in proportion to the number of devices while keeping data-parallel efficiency. Implemented in DeepSpeed, it enabled training of models into the hundreds of billions of parameters on existing clusters.