Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

9 records

0012018Data, Corpora & Tokenization

Software release

SentencePiece

Taku Kudo, John Richardson

SentencePiece is a tokenizer that learns a subword vocabulary straight from raw text, without needing a separate word-splitting step that most earlier pipelines assumed. It escapes whitespace as a normal symbol (the underscore marker) so that tokenizing and detokenizing are exactly reversible, which matters for languages like Japanese or Chinese that do not put spaces between words. It supports both byte-pair-encoding and unigram-language-model segmentation and ships as a library with fixed, serializable models so the same text always maps to the same tokens. This standardized, language-agnostic tokenization step is now a default component in many multilingual and non-English NLP systems.

Source availableDifficulty 4/10Verified
Read record Primary source
0022022Data, Corpora & Tokenization

Peer reviewed

Deduplicating Training Data Makes Language Models Better

Katherine Lee, Daphne Ippolito, Andrew Nystrom, et al.

The authors measured duplication in common pretraining datasets and found that many sequences appear many times, including substantial overlap between training and evaluation sets. They built two deduplication tools: an exact-substring method using suffix arrays to find long repeated spans, and an approximate document-level method using MinHash to catch near-duplicates. Training on the deduplicated data reduced how often models regurgitated memorized text verbatim, gave more trustworthy evaluation numbers by removing leaked test examples, and required fewer training steps to reach a given accuracy. The paper made corpus deduplication a standard preprocessing step for language-model training.

UnknownDifficulty 4/10Verified
Read record Primary source
0032021Data, Corpora & Tokenization

Dataset paper

The Pile

Leo Gao, Stella Biderman, Sid Black, et al.

EleutherAI combined 22 sources spanning academic writing, code, books, legal and other domains into a single 800GB corpus, deliberately weighting high-quality and specialized text rather than relying only on filtered Common Crawl. They documented composition, per-source weighting, and preprocessing, and showed models trained on this mixture improved on domain-specific evaluation relative to web-only data. It became a widely used training and benchmarking corpus for open models such as GPT-Neo and GPT-J.

UnknownDifficulty 4/10Verified
Read record Primary source
0042009Data, Corpora & Tokenization

Peer reviewed

The Unreasonable Effectiveness of Data

Alon Halevy, Peter Norvig, Fernando Pereira

Drawing on web-scale examples, the authors contend that unreasonably large corpora let comparatively simple methods succeed where elaborate ones with little data fail, and urge researchers to embrace data. Written years before deep learning’s dominance, it foreshadowed the data-centric logic later formalized by scaling laws. It is a conceptual ancestor of the scaling era.

UnknownDifficulty 4/10Verified
Read record Primary source
0052023Data, Corpora & Tokenization

Dataset paper

The RefinedWeb Dataset for Falcon LLM

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al.

The authors built a pipeline (MacroData Refinement) that applies URL filtering, language identification, trafilatura-based text extraction, quality heuristics, and both fuzzy and exact deduplication to Common Crawl at scale. From this they released RefinedWeb, a five-trillion-token web-only corpus, and trained Falcon models that matched or exceeded models trained on curated mixtures. This demonstrated that scale plus rigorous cleaning of raw web data can substitute for hand-picked high-quality sources, and provided a large open dataset for the community.

UnknownDifficulty 5/10Verified
Read record Primary source
0062023Data, Corpora & Tokenization

Technical report

Textbooks Are All You Need (phi)

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, et al.

The authors assembled a filtered set of high-educational-value code from the web plus GPT-generated textbook-style text and problem/solution exercises, then trained phi-1, a 1.3B model, on only about 7 billion tokens. Despite being orders of magnitude smaller in data and parameters than contemporaries, phi-1 reached strong pass@1 on HumanEval and MBPP. This provided evidence that carefully curated, instruction-dense data can dramatically improve sample efficiency, seeding the small-high-quality-data line of work.

UnknownDifficulty 5/10Verified
Read record Primary source
0072024Data, Corpora & Tokenization

Peer reviewed

The Curse of Recursion / Model Collapse

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson

The authors trained models on data produced by earlier model generations and observed that each generation's outputs narrow toward the mean, losing rare events and low-probability tails until later models converge to a degenerate distribution. They gave a theoretical account attributing this to compounding statistical approximation, functional expressivity, and sampling errors across generations. The finding warned that as AI-generated text saturates the web, indiscriminate training on it degrades future models, making access to genuine human-produced data increasingly valuable.

UnknownDifficulty 5/10Verified
Read record Primary source
0082018Data, Corpora & Tokenization

Peer reviewed

Subword Regularization / Unigram LM Tokenization

Taku Kudo

Standard subword methods like BPE give one deterministic segmentation per word, so the model never sees alternative splits. This work defines a unigram LM over subwords that can yield multiple probable segmentations and samples among them each epoch, exposing the model to varied tokenizations of identical text. Acting as data augmentation, it improved neural machine translation accuracy especially on low-resource and noisy settings, and the unigram tokenizer became a widely used alternative to BPE (shipped in SentencePiece).

UnknownDifficulty 5/10Verified
Read record Primary source
0092023Data, Corpora & Tokenization

Peer reviewed

DoReMi: Optimizing Data Mixtures with Group DRO

Sang Michael Xie, Hieu Pham, Xuanyi Dong, et al.

The method first trains a small reference model, then trains a second small proxy model with Group Distributionally Robust Optimization that raises the sampling weight of domains where the proxy has the largest excess loss over the reference. The resulting domain weights are reused to sample data for a much larger model. This removed the need to hand-tune or grid-search corpus mixtures at full scale, and reached target accuracy in fewer training steps by reweighting domains like The Pile rather than using their default proportions.

UnknownDifficulty 6/10Verified
Read record Primary source