Decoder-Only Language Models · 2019
Language Models are Unsupervised Multitask Learners (GPT-2)
Demonstrated that scaling a decoder-only language model and its training corpus lets it perform many NLP tasks with no gradient updates, purely by conditioning on a prompt.
Editorial record
Plain-language summary
GPT-2 trains a 1.5B-parameter Transformer on WebText, a large corpus scraped from outbound Reddit links, using the same next-token objective as GPT-1. The paper shows the model handles reading comprehension, translation, summarization, and question answering in a zero-shot setting when the task is phrased as text, without any task-specific training. This reframed NLP tasks as special cases of language modeling and gave early evidence that capability scales with model and data size.
Knowledge graph
Relationships
Antecedents
ScalesEvidence: Direct
Language Models are Few-Shot Learners (GPT-3)
Scaled to 175B; few-shot in-context learning emerges
P-012 paper
Descendants
ScalesEvidence: Direct
Improving Language Understanding by Generative Pre-Training (GPT)
Same generative-pretraining recipe scaled to 1.5B
P-011 report
Source record
Provenance
- Record ID
- P-011
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.