Understanding the Difficulty of Training Deep Nets (Xavier/He init)

Editorial record

Plain-language summary

The paper measures how the variance of activations and back-propagated gradients changes from layer to layer under different setups, showing that poor scaling makes deep networks with saturating activations (like sigmoid) hard to train. It proposes initializing weights with a variance set from the number of incoming and outgoing connections so that signals neither vanish nor blow up as they pass through the layers. This initialization let deeper networks converge more reliably and became a default starting point for many architectures.