Transformer Architecture · 2020
GLU Variants Improve Transformer (SwiGLU)
Showed that replacing the transformer feed-forward layer's ReLU with gated linear unit variants, especially SwiGLU, yields better quality at matched compute.
Editorial record
Plain-language summary
The paper swaps the standard two-matrix ReLU feed-forward block for GLU variants, where one linear projection is elementwise-multiplied by a gated (sigmoid, GELU, or Swish) version of another projection. In controlled pretraining and fine-tuning comparisons, these gated blocks lower perplexity and improve downstream scores over the plain ReLU feed-forward. SwiGLU became the standard feed-forward design in PaLM, LLaMA, and most subsequent large models.
Knowledge graph
Relationships
Descendants
ImprovesEvidence: Direct
Attention Is All You Need
SwiGLU replaces ReLU FFN in Transformer
P-004 paper
GeneralizesEvidence: Strongly supported
Rectified Linear Units (ReLU)
ReLU is the parent of gated SwiGLU activations
P-004
Source record
Provenance
- Record ID
- P-004
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2002.05202
- arXiv:2002.05202
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.