Improving Language Understanding by Generative Pre-Training (GPT)

Editorial record

Plain-language summary

GPT-1 pre-trains a left-to-right Transformer on a large corpus of books to predict the next token, then adapts to each downstream task with a small supervised fine-tuning step and task-specific input formatting. This two-stage recipe removed the need to hand-design a separate model per task and to rely on scarce labeled data. It improved results on entailment, question answering, semantic similarity, and classification, establishing generative pre-training plus fine-tuning as a general NLP method.