Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

9 records

0012022Scaling Laws & Compute

Peer reviewed

Training Compute-Optimal Large Language Models (Chinchilla)

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al.

DeepMind trained over 400 models at varied size and token counts and re-estimated the compute-optimal frontier, finding that for a given compute budget parameters and data should grow together at about a 1:1 ratio rather than favoring size. To demonstrate it they trained Chinchilla, a 70B model on 1.4 trillion tokens, which outperformed the 280B Gopher and other larger models while being cheaper to run at inference. The result redirected the field toward training smaller models on far more data.

UnknownDifficulty 6/10Verified
Read record Primary source
0022020Scaling Laws & Compute

Preprint

Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al.

The authors trained many Transformer language models across orders of magnitude in parameters, data, and compute, then fit power laws to the loss curves. They showed loss scales predictably with each factor when the others are not bottlenecked, and that within their observed range larger models are more sample-efficient, so given fixed compute it was better to train very large models on comparatively less data and stop early. This gave labs a quantitative basis to forecast returns from more compute and allocate budget before committing to a run.

UnknownDifficulty 6/10Verified
Read record Primary source
0032019Scaling Laws & Compute

Essay

The Bitter Lesson

Richard Sutton

Sutton observes that in game-playing, speech, and vision, hand-crafted domain knowledge gave early wins but was eventually overtaken by search and learning methods that scale with compute. The uncomfortable conclusion is that human insight matters less than the ability to exploit growing computation. The essay became a rallying text for scaling-first research.

UnknownDifficulty 3/10Verified
Read record Primary source
0042020Scaling Laws & Compute

Essay

The Scaling Hypothesis

Gwern Branwen

Written after GPT-3, the essay collects the evidence that loss and many capabilities improve predictably with size, and contends that the field had underrated how far pure scaling would go. It frames scaling as a hypothesis to be taken seriously rather than a curiosity. It is one of the defining statements of the scaling-era worldview and its stance is debated.

UnknownDifficulty 4/10Verified
Read record Primary source
0052022Scaling Laws & Compute

Peer reviewed

Emergent Abilities of Large Language Models

Jason Wei, Yi Tay, Rishi Bommasani, et al.

The authors survey tasks where accuracy stays near chance for small models and then rises sharply beyond a certain parameter or compute scale, calling these emergent abilities. They catalog examples across few-shot prompting and augmented-prompting settings, showing the jumps are not evident from the performance trend of smaller models. The claim shaped how the field reasoned about scaling, suggesting some capabilities cannot be forecast and appear only after crossing a size threshold.

UnknownDifficulty 5/10Verified
Read record Primary source
0062023Scaling Laws & Compute

Peer reviewed

Are Emergent Abilities of LLMs a Mirage?

Rylan Schaeffer, Brando Miranda, Sanmi Koyejo

Schaeffer et al. show that metrics like exact-match or multiple-choice accuracy are harsh and nonlinear: a model's underlying per-token performance can improve smoothly while the reported score stays flat then spikes once the whole answer becomes correct. Re-scoring the same models and tasks with continuous or partial-credit metrics converts many reported emergent curves into steady, extrapolable trends. The work reframes much claimed emergence as a measurement choice, cautioning that metric selection, not new capability, can manufacture the appearance of a threshold.

UnknownDifficulty 5/10Verified
Read record Primary source
0072023Scaling Laws & Compute

Peer reviewed

Scaling Data-Constrained Language Models

Niklas Muennighoff, Alexander M. Rush, Boaz Barak, et al.

The authors trained hundreds of models while holding the amount of unique text fixed and varying how many times it was repeated, then fit scaling laws that treat repeated tokens as worth less than fresh ones. They found that repeating data for up to about four epochs yields loss nearly identical to using that much new data, with returns decaying quickly after roughly 16 epochs. This gave data-limited teams a principled way to allocate compute between more epochs and more parameters, and quantified when scraping or generating more data stops helping.

UnknownDifficulty 6/10Verified
Read record Primary source
0082024Scaling Laws & Compute

Vision essay

Situational Awareness: The Decade Ahead

Leopold Aschenbrenner

The long essay projects the straight lines on graphs of training compute and efficiency forward, argues that the gap from current models to human-level research assistants is a few more orders of magnitude, and discusses security and governance consequences. It is a prediction and advocacy piece, not an empirical result, and its timelines are contested. It captured a strand of frontier-lab thinking in 2024.

UnknownDifficulty 3/10Verified
Read record Primary source
0092018Scaling Laws & Compute

Technical report

AI and Compute

Dario Amodei, Danny Hernandez

The analysis charts training compute across landmark systems and finds a doubling time far faster than Moore’s law over the studied period. It made the compute-growth trend legible and widely cited as evidence that scaling, not just architecture, was driving progress. It is an early empirical anchor for the scaling narrative.

UnknownDifficulty 3/10Verified
Read record Primary source