Data, Corpora & Tokenization · 2023
The RefinedWeb Dataset for Falcon LLM
It showed that aggressive filtering and deduplication of Common Crawl alone can produce web text good enough to train competitive LLMs, challenging the assumption that curated sources like books and Wikipedia are required.
Editorial record
Plain-language summary
The authors built a pipeline (MacroData Refinement) that applies URL filtering, language identification, trafilatura-based text extraction, quality heuristics, and both fuzzy and exact deduplication to Common Crawl at scale. From this they released RefinedWeb, a five-trillion-token web-only corpus, and trained Falcon models that matched or exceeded models trained on curated mixtures. This demonstrated that scale plus rigorous cleaning of raw web data can substitute for hand-picked high-quality sources, and provided a large open dataset for the community.
Knowledge graph
Relationships
Descendants
ExtendsEvidence: Strongly supported
The Pile
RefinedWeb extends open-corpus practice with hard filtering
P-124
Source record
Provenance
- Record ID
- P-124
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2306.01116
- arXiv:2306.01116
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.