Encoders · 2020
ELECTRA: Pre-training Text Encoders as Discriminators
ELECTRA trains an encoder with replaced-token detection, a discriminator that classifies every token as original or generator-substituted, extracting learning signal from all positions instead of only the ~15% masked ones and cutting pretraining compute.
Editorial record
Plain-language summary
Rather than masking tokens and predicting them, ELECTRA uses a small generator to swap some tokens for plausible alternatives, then trains the main model to decide, for every token, whether it was replaced. Because the loss covers all positions rather than the small masked subset, each training step yields more signal per example. This let ELECTRA match or exceed BERT-scale accuracy using substantially less pretraining compute, making strong encoders reachable on smaller budgets.
Knowledge graph
Relationships
Descendants
ImprovesEvidence: Direct
BERT: Pre-training of Deep Bidirectional Transformers
Replaced-token-detection objective improves sample efficiency
P-016 paper
Source record
Provenance
- Record ID
- P-016
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2003.10555
- arXiv:2003.10555
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.