Data, Corpora & Tokenization · 2021
The Pile
Assembled an 800GB English text corpus from 22 curated, diverse sources, giving open language-model training a large, documented, multi-domain dataset instead of ad hoc web scrapes.
Editorial record
Plain-language summary
EleutherAI combined 22 sources spanning academic writing, code, books, legal and other domains into a single 800GB corpus, deliberately weighting high-quality and specialized text rather than relying only on filtered Common Crawl. They documented composition, per-source weighting, and preprocessing, and showed models trained on this mixture improved on domain-specific evaluation relative to web-only data. It became a widely used training and benchmarking corpus for open models such as GPT-Neo and GPT-J.
Knowledge graph
Relationships
Antecedents
ExtendsEvidence: Strongly supported
The RefinedWeb Dataset for Falcon LLM
RefinedWeb extends open-corpus practice with hard filtering
P-124
Source record
Provenance
- Record ID
- P-123
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2101.00027
- arXiv:2101.00027
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.