Transformer Architecture · 2019
Root Mean Square Layer Normalization (RMSNorm)
Introduced RMSNorm, which normalizes activations by their root-mean-square alone and drops the mean-centering and bias of LayerNorm, cutting normalization cost while keeping training stability.
Editorial record
Plain-language summary
RMSNorm rescales each activation vector by its root-mean-square magnitude and applies a learned gain, skipping the mean-subtraction step that standard LayerNorm performs. The authors argue the re-centering in LayerNorm contributes little and that re-scaling is what actually stabilizes training. The result is fewer operations and less compute per layer at comparable accuracy, which is why it replaced LayerNorm in LLaMA and many later transformer stacks.
Knowledge graph
Relationships
Descendants
Makes efficientEvidence: Direct
Attention Is All You Need
RMSNorm simplifies LayerNorm used in Transformer
P-003 paper
Source record
Provenance
- Record ID
- P-003
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1910.07467
- arXiv:1910.07467
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.