Alignment & Preference Learning · 2022
Constitutional AI: Harmlessness from AI Feedback (RLAIF)
Replaced most human-labeled harmlessness feedback with model-generated feedback governed by a written set of principles, removing the need for large volumes of human labels on harmful content to train a harmless assistant.
Editorial record
Plain-language summary
In a supervised phase the model critiques and revises its own responses against a short list of natural-language principles (a 'constitution'), and in an RL phase a model rather than humans ranks response pairs for harmlessness to train the preference model (RLAIF). This let the assistant refuse or push back on harmful requests while explaining its reasoning instead of giving evasive non-answers. It cut human labeling of toxic content and made the value targets explicit and editable as written rules.
Knowledge graph
Relationships
Descendants
ExtendsEvidence: Direct
InstructGPT: Training LMs to Follow Instructions with Human Feedback
Constitutional AI replaces human feedback with AI feedback
P-204
Source record
Provenance
- Record ID
- P-204
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2212.08073
- arXiv:2212.08073
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.