Data, Corpora & Tokenization · 2023
Textbooks Are All You Need (phi)
It showed that training a small code model on a small corpus of textbook-quality and synthetically generated exercises yields coding ability far above what its parameter and token count would predict, shifting emphasis from data quantity to data quality.
Editorial record
Plain-language summary
The authors assembled a filtered set of high-educational-value code from the web plus GPT-generated textbook-style text and problem/solution exercises, then trained phi-1, a 1.3B model, on only about 7 billion tokens. Despite being orders of magnitude smaller in data and parameters than contemporaries, phi-1 reached strong pass@1 on HumanEval and MBPP. This provided evidence that carefully curated, instruction-dense data can dramatically improve sample efficiency, seeding the small-high-quality-data line of work.
Knowledge graph
Relationships
Antecedents
ChallengesEvidence: Strongly supported
The Curse of Recursion / Model Collapse
Model collapse bounds synthetic-data optimism
P-126
Source record
Provenance
- Record ID
- P-125
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2306.11644
- arXiv:2306.11644
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.