Learning Long-Term Dependencies is Difficult (vanishing gradients)

Editorial record

Plain-language summary

The paper showed mathematically that for a recurrent network to store information robustly over time its dynamics must contract, but contraction causes error gradients passed backward through many time steps to shrink toward zero, so distant past inputs contribute almost nothing to weight updates. When the dynamics instead expand, gradients blow up. This trade-off means standard gradient descent cannot reliably link causes and effects separated by many steps. The analysis motivated later remedies such as gating (LSTM, GRU), gradient clipping, and better initialization.