Encoders · 2019
ALBERT: A Lite BERT
ALBERT cuts BERT's parameter count through factorized embedding parameterization and cross-layer parameter sharing, and swaps next-sentence prediction for a sentence-order objective, letting larger-capacity configurations train within the same memory budget.
Editorial record
Plain-language summary
ALBERT decouples the vocabulary embedding size from the hidden size (factorizing that large matrix) and shares the same parameters across all Transformer layers, drastically reducing the model's parameter footprint. It also replaces BERT's next-sentence prediction with a sentence-order prediction task that forces the model to learn inter-sentence coherence rather than topic overlap. Together these let ALBERT scale hidden dimensions and depth without a proportional memory blowup, reaching stronger results than BERT-large at a fraction of the stored parameters.
Knowledge graph
Relationships
Descendants
Makes efficientEvidence: Direct
BERT: Pre-training of Deep Bidirectional Transformers
Cross-layer parameter sharing and factorized embeddings
P-015 abstract
Source record
Provenance
- Record ID
- P-015
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1909.11942
- arXiv:1909.11942
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.