GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism

Editorial record

Plain-language summary

The model layers are divided into consecutive stages placed on separate accelerators, and each minibatch is broken into micro-batches that flow through the stages so multiple devices compute at once instead of waiting idle. Gradients are accumulated across micro-batches and applied synchronously, so the result is numerically identical to non-pipelined training. Combined with activation recomputation, this let models with billions of parameters be trained across devices that individually could not hold them, at near-linear throughput scaling.