Learning to Summarize from Human Feedback

Editorial record

Plain-language summary

Instead of training the model to imitate reference summaries, the authors collected human judgments of which of two summaries was better, trained a reward model to predict those judgments, and then used reinforcement learning to make the summarizer score highly on that reward. The resulting summaries were preferred by people over both the reference summaries and models trained by supervised imitation. This work is the practical template for reinforcement learning from human feedback (RLHF) that later underpinned instruction-following chat models.