Rectified Linear Units (ReLU)

Editorial record

Plain-language summary

A ReLU passes positive inputs through unchanged and outputs zero for negative inputs, so its gradient is a constant one wherever the unit is active instead of shrinking toward zero the way sigmoid and tanh gradients do. This keeps gradients flowing through many layers and produces sparse activations that are cheap to compute. Using ReLUs let deep networks train faster and reach lower error, and it became the default activation for most feedforward and convolutional networks.