PipeDream: Generalized Pipeline Parallelism for DNN Training

Editorial record

Plain-language summary

Rather than draining the pipeline between minibatches, PipeDream uses a one-forward-one-backward schedule that overlaps the forward and backward work of different minibatches to keep all stages busy. Because a minibatch's backward pass then sees newer weights than its forward pass, it stashes the weight version used in the forward pass and reuses it in the backward pass so gradients stay consistent. An automatic profiler partitions layers across stages to balance compute and communication, cutting time-to-target-accuracy relative to data-parallel and flush-based pipeline training.