Neural Foundations · 2016
Layer Normalization
Introduced layer normalization, which normalizes activations across the features of a single training example rather than across the batch, removing batch-size dependence and making normalization usable in recurrent networks and at inference on single examples.
Editorial record
Plain-language summary
The method computes each layer's normalization statistics over the units within one example, so behavior is identical during training and testing and does not depend on batch composition. This made it effective for RNNs and sequence models where batch normalization worked poorly, stabilizing and speeding up training. Layer normalization became standard in Transformer architectures.
Knowledge graph
Relationships
Antecedents
Depends onEvidence: Direct
Attention Is All You Need
LayerNorm applied around each sublayer
P-001 Sec 3.1
Descendants
GeneralizesEvidence: Strongly supported
Batch Normalization
LayerNorm adapts BatchNorm for sequences
A-010
Source record
Provenance
- Record ID
- A-010
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1607.06450
- arXiv:1607.06450
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.