Alignment & Preference Learning · 2022
InstructGPT: Training LMs to Follow Instructions with Human Feedback
Applied reinforcement learning from human feedback to GPT-3 so a much smaller aligned model followed user intent better than the raw base model, closing the gap between next-token pretraining and doing what a user actually asked.
Editorial record
Plain-language summary
Labelers wrote demonstrations to supervise-fine-tune GPT-3, then ranked model outputs to train a reward model, and PPO optimized the policy against that reward. Outputs from the 1.3B InstructGPT model were preferred to the 175B GPT-3's despite being about 100x smaller, with gains in truthfulness and reductions in toxic generation. The three-stage SFT-then-reward-model-then-PPO recipe became the standard alignment pipeline behind instruction-following chat models.
Knowledge graph
Relationships
Antecedents
ExtendsEvidence: Direct
Constitutional AI: Harmlessness from AI Feedback (RLAIF)
Constitutional AI replaces human feedback with AI feedback
P-204
ReplacesEvidence: Direct
Direct Preference Optimization (DPO)
DPO removes reward model and PPO from RLHF objective
P-205
ChallengesEvidence: Direct
Reward Model Overoptimization / Sycophancy
Overoptimization and sycophancy are RLHF failure modes
P-207
Descendants
ExtendsEvidence: Direct
Learning to Summarize from Human Feedback
InstructGPT extends RLHF-for-text to instruction following
P-202
Depends onEvidence: Strongly supported
Finetuned Language Models Are Zero-Shot Learners (FLAN) / Self-Instruct
SFT/instruction data is stage 1 of RLHF
P-202
Source record
Provenance
- Record ID
- P-202
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2203.02155
- arXiv:2203.02155
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.