Evaluation & Benchmarks · 2021
MMLU: Massive Multitask Language Understanding
A 57-subject multiple-choice benchmark spanning STEM, humanities, social sciences, and professional exams that measures broad pretrained knowledge in zero- and few-shot settings, removing reliance on narrow single-task metrics for judging general capability.
Editorial record
Plain-language summary
The benchmark collects roughly 16,000 exam-style questions at difficulties from high school to professional level and scores models without task-specific training. At release most models scored near the 25% random-chance baseline while the largest GPT-3 reached about 44%, with especially weak and poorly calibrated performance on subjects like law and morality. It became a standard headline number for comparing the general knowledge of frontier models.
Knowledge graph
Relationships
Antecedents
ChallengesEvidence: Direct
The Evaluation Crisis (contamination/saturation/judge-bias)
MMLU saturation/contamination feeds eval crisis
P-449
Source record
Provenance
- Record ID
- P-440
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2009.03300
- arXiv:2009.03300
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.