Batch Normalization

Editorial record

Plain-language summary

For each mini-batch, the method normalizes a layer's pre-activation values to zero mean and unit variance, then applies a learned scale and shift so the layer can still represent what it needs. This keeps the distribution of inputs to each layer more stable as the weights below it change, which allows much higher learning rates and less sensitivity to initialization. It sped up training substantially, added a mild regularizing effect, and became a standard component of deep convolutional networks.