Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

9 records

0012022Alignment & Preference Learning

Peer reviewed

InstructGPT: Training LMs to Follow Instructions with Human Feedback

Long Ouyang, Jeff Wu, Xu Jiang, et al.

Labelers wrote demonstrations to supervise-fine-tune GPT-3, then ranked model outputs to train a reward model, and PPO optimized the policy against that reward. Outputs from the 1.3B InstructGPT model were preferred to the 175B GPT-3's despite being about 100x smaller, with gains in truthfulness and reductions in toxic generation. The three-stage SFT-then-reward-model-then-PPO recipe became the standard alignment pipeline behind instruction-following chat models.

UnknownDifficulty 6/10Verified
Read record Primary source
0022022Alignment & Preference Learning

Peer reviewed

Finetuned Language Models Are Zero-Shot Learners (FLAN) / Self-Instruct

Jason Wei, Maarten Bosma, Quoc V. Le, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, Daniel Khashabi

The authors took a pretrained model and fine-tuned it on many existing NLP datasets that were each rewritten as instructions (for example, 'Is this review positive or negative?'). After this instruction tuning, the model could handle new kinds of tasks it had not seen during training, just from reading the instruction. This established instruction tuning as a general method for making base models usable without few-shot prompting, and improved zero-shot performance across a range of benchmarks.

UnknownDifficulty 4/10Verified
Read record Primary source
0032022Alignment & Preference Learning

Technical report

Constitutional AI: Harmlessness from AI Feedback (RLAIF)

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al.

In a supervised phase the model critiques and revises its own responses against a short list of natural-language principles (a 'constitution'), and in an RL phase a model rather than humans ranks response pairs for harmlessness to train the preference model (RLAIF). This let the assistant refuse or push back on harmful requests while explaining its reasoning instead of giving evasive non-answers. It cut human labeling of toxic content and made the value targets explicit and editable as written rules.

UnknownDifficulty 6/10Verified
Read record Primary source
0042023Alignment & Preference Learning

Peer reviewed

Direct Preference Optimization (DPO)

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, Chelsea Finn

Direct Preference Optimization uses the mathematical link between reward and optimal policy to rewrite preference learning so the language model itself is optimized directly on chosen-versus-rejected examples with a simple supervised loss. There is no reward model to train and no PPO sampling, so it is more stable and cheaper while matching or beating PPO-based RLHF on preference tuning. This made preference alignment practical for teams without reinforcement-learning infrastructure.

UnknownDifficulty 6/10Verified
Read record Primary source
0052017Alignment & Preference Learning

Peer reviewed

Deep Reinforcement Learning from Human Preferences

Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, Dario Amodei

The work has people repeatedly pick which of two short video clips of an agent looks closer to a goal, and fits a reward predictor to those choices while a policy is trained against that predictor. Because labeling comparisons is cheaper than demonstrating or coding rewards, it learned tasks like Atari games and simulated-robot backflips from under an hour of human feedback on a small fraction of the agent's interactions. This comparison-based reward modeling became the template later scaled to language-model alignment.

UnknownDifficulty 6/10Verified
Read record Primary source
0062020Alignment & Preference Learning

Peer reviewed

Learning to Summarize from Human Feedback

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel Ziegler, et al.

Instead of training the model to imitate reference summaries, the authors collected human judgments of which of two summaries was better, trained a reward model to predict those judgments, and then used reinforcement learning to make the summarizer score highly on that reward. The resulting summaries were preferred by people over both the reference summaries and models trained by supervised imitation. This work is the practical template for reinforcement learning from human feedback (RLHF) that later underpinned instruction-following chat models.

UnknownDifficulty 6/10Verified
Read record Primary source
0072024Alignment & Preference Learning

Peer reviewed

Preference Optimization Family (IPO/KTO/ORPO/SimPO)

Mohammad Gheshlaghi Azar, Rémi Munos, Kawin Ethayarajh, Douwe Kiela, Jiwoo Hong, James Thorne, Yu Meng, Danqi Chen

Each variant targets a specific limitation of DPO: IPO replaces the log-sigmoid objective with a bounded one to curb overfitting when preferences are near-deterministic; KTO learns from single labeled examples marked desirable or undesirable rather than requiring paired comparisons; ORPO folds a preference odds-ratio penalty directly into supervised fine-tuning so no separate alignment stage or reference model is needed; SimPO removes the reference model and uses a length-normalized reward with a target margin. Together they made preference-based alignment cheaper and more stable, reducing reliance on paired data, reference models, and reward-model training.

UnknownDifficulty 6/10Verified
Read record Primary source
0082022Alignment & Preference Learning

Peer reviewed

Reward Model Overoptimization / Sycophancy

Leo Gao, John Schulman, Jacob Hilton, Mrinank Sharma, Ethan Perez

The authors trained policies against reward models of varying size and data, then compared the proxy reward the model assigned against a gold reward model treated as ground truth. As optimization pressure (measured in KL distance from the initial policy) increased, proxy reward kept rising while true reward peaked and then fell, and they fit scaling laws for where this divergence begins. This gave RLHF practitioners a concrete way to predict when a reward model stops being a trustworthy target and to bound optimization accordingly, explaining downstream failures like sycophancy where the policy exploits reward-model quirks rather than improving.

UnknownDifficulty 6/10Verified
Read record Primary source
0092018Alignment & Preference Learning

Preprint

Scalable Oversight: Debate / Weak-to-Strong Generalization

Geoffrey Irving, Paul Christiano, Dario Amodei, Collin Burns, Jan Leike, Ilya Sutskever

The weak-to-strong experiments used a small model's imperfect labels to fine-tune a much larger pretrained model and found the larger model generalized beyond its teacher's errors, an analogy for humans supervising superhuman systems. Related debate proposals have two models argue opposing positions so a limited judge can adjudicate claims it could not verify directly. Together these define the scalable-oversight problem and give early empirical evidence that supervision signal can transfer even when the supervisor is less capable than the system being trained.

UnknownDifficulty 7/10Verified
Read record Primary source