Retrieval & Memory · 2020
REALM: Retrieval-Augmented LM Pre-Training
REALM introduced end-to-end pretraining of a language model jointly with a neural retriever over a text corpus, removing the need to store all world knowledge implicitly in model parameters.
Editorial record
Plain-language summary
REALM augments masked language-model pretraining with a learned retriever that fetches relevant documents from a corpus and conditions predictions on them, backpropagating the language-modeling signal through the retrieval step so the retriever learns which passages help. This let a relatively small model consult an external, updatable knowledge store instead of memorizing facts in its weights, improving open-domain question answering and making the knowledge source inspectable and swappable.
Knowledge graph
Relationships
Antecedents
Depends onEvidence: Direct
Retrieval-Augmented Generation (RAG)
RAG builds on retrieval-augmented LM idea
P-322
Source record
Provenance
- Record ID
- P-321
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2002.08909
- arXiv:2002.08909
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.