Encoders · 2018
BERT: Pre-training of Deep Bidirectional Transformers
Introduced BERT, a Transformer encoder pre-trained with masked-language-modeling and next-sentence prediction to produce deeply bidirectional representations usable across tasks via light fine-tuning.
Editorial record
Plain-language summary
BERT pre-trains an encoder by masking random tokens and predicting them from both left and right context, unlike the left-to-right models of the GPT line, plus a next-sentence-prediction objective. The resulting representations condition each token on the full surrounding sentence, and a single added output layer fine-tunes the model for classification, tagging, or span-based question answering. It set new results on GLUE and SQuAD and became the standard encoder for understanding-oriented NLP tasks.
Knowledge graph
Relationships
Antecedents
ImprovesEvidence: Direct
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Corrective ablation: drop NSP train longer on more data
P-014 abstract
Makes efficientEvidence: Direct
ALBERT: A Lite BERT
Cross-layer parameter sharing and factorized embeddings
P-015 abstract
ImprovesEvidence: Direct
ELECTRA: Pre-training Text Encoders as Discriminators
Replaced-token-detection objective improves sample efficiency
P-016 paper
ImprovesEvidence: Direct
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Disentangled content/position attention
P-024 paper
Applies toEvidence: Direct
Dense Passage Retrieval (DPR)
DPR uses BERT dual encoders
P-320
Applies toEvidence: Direct
Contrastive Text Embeddings (Sentence-BERT/SimCSE/E5/BGE)
Sentence-BERT adapts BERT for embeddings
P-325
CombinesEvidence: Strongly supported
GLM: General Language Model Pretraining with Autoregressive Blank Infilling / GLM-130B
GLM blank-infilling incorporates BERT-style bidirectional masked prediction
GLM 2103.10360
Descendants
Applies toEvidence: Direct
Attention Is All You Need
Encoder-only masked LM built on Transformer encoder
P-013 architecture
GeneralizesEvidence: Strongly supported
Bidirectional Recurrent Neural Networks
Bidirectionality realized by BERT
P-013
Source record
Provenance
- Record ID
- P-013
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1810.04805
- arXiv:1810.04805
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.