Encoders · 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach
It showed that BERT was substantially undertrained and that with more data, longer training, larger batches, removal of the next-sentence-prediction objective, and dynamic masking, the same architecture reaches much higher accuracy, establishing that pretraining recipe choices, not architecture, drove most of the remaining gains.
Editorial record
Plain-language summary
RoBERTa is a careful re-examination of how BERT was trained rather than a new model architecture. The authors trained longer on about ten times more text, used bigger batches, generated the masking pattern fresh each time an example is seen (dynamic masking), dropped the next-sentence-prediction task, and trained on full-length sentence sequences. These changes alone pushed performance above the original BERT and matched or beat later models on GLUE, SQuAD, and RACE. The main lesson was methodological: much of what looked like architectural progress was actually the result of undertraining, and controlled comparisons require fixing the training budget.
Knowledge graph
Relationships
Descendants
ImprovesEvidence: Direct
BERT: Pre-training of Deep Bidirectional Transformers
Corrective ablation: drop NSP train longer on more data
P-014 abstract
Source record
Provenance
- Record ID
- P-014
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1907.11692
- arXiv:1907.11692
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.