Transformer Architecture · 2020
On Layer Normalization in the Transformer Architecture (pre-norm)
It showed analytically and empirically that placing layer normalization inside the residual branch (Pre-LN) instead of after it (Post-LN) keeps gradients well-scaled at initialization, removing the need for the learning-rate warm-up stage that Post-LN Transformers required for stable training.
Editorial record
Plain-language summary
The paper studies where the layer-normalization step sits relative to the residual connection in a Transformer block. In the original Post-LN design, gradients near the output layer are large at initialization, so training diverges unless you start with a small learning rate and slowly warm it up. By moving the normalization to the input of each sub-layer (Pre-LN), gradients become bounded and roughly uniform across depth, so models train stably without warm-up, tolerate larger learning rates, and converge faster. This made deep Transformers easier and cheaper to train and is why most later large models adopt the Pre-LN arrangement.
Knowledge graph
Relationships
Descendants
Makes efficientEvidence: Direct
Attention Is All You Need
Pre-norm placement stabilizes deep Transformer training
P-007 paper
Source record
Provenance
- Record ID
- P-007
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2002.04745
- arXiv:2002.04745
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.