Decoder-Only Language Models · 2020
Language Models are Few-Shot Learners (GPT-3)
Showed that scaling an autoregressive language model to 175B parameters yields in-context few-shot learning, where tasks are specified through examples in the prompt rather than by fine-tuning weights.
Editorial record
Plain-language summary
GPT-3 trains a 175-billion-parameter Transformer on a filtered Common Crawl plus other corpora, keeping the next-token objective but scaling roughly 100x over GPT-2. Given a natural-language instruction and a handful of demonstrations in its context window, it performs translation, question answering, arithmetic, and other tasks without weight updates, with accuracy generally rising as more examples are shown. This removed the per-task fine-tuning and labeled-data requirement for many uses and made prompting the primary interface to large models.
Knowledge graph
Relationships
Antecedents
Provides evidence forEvidence: Direct
Scaling Laws for Neural Language Models
GPT-3-era models motivate the scaling-law study
P-100
EnablesEvidence: Direct
Chain-of-Thought Prompting Elicits Reasoning
Chain-of-thought emerges only at scale
P-220
Depends onEvidence: Direct
WebGPT: Browser-assisted Question-answering with Human Feedback
WebGPT fine-tunes GPT-3 to browse and answer with citations
P-520
Applies toEvidence: Direct
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan)
SayCan grounds a language model in a robot value function of affordances
P-521
Descendants
ScalesEvidence: Direct
Language Models are Unsupervised Multitask Learners (GPT-2)
Scaled to 175B; few-shot in-context learning emerges
P-012 paper
Provides evidence forEvidence: Strongly supported
In-context Learning and Induction Heads
Induction heads give a mechanistic account of in-context learning
P-502
ChallengesEvidence: Strongly supported
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
Critiques the scale-first paradigm on cost bias and environmental grounds
book
Source record
Provenance
- Record ID
- P-012
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2005.14165
- arXiv:2005.14165
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.