Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)

Editorial record

Plain-language summary

T5 uses an encoder-decoder Transformer trained with a span-corruption denoising objective and represents each task, including ones with numeric or label outputs, as producing a target string from an input string. The paper runs a controlled comparison of architectures, objectives, corpora, and transfer strategies, and pre-trains on the C4 Common Crawl corpus the authors assembled and released. The unified format removed per-task output heads and enabled systematic study of what drives transfer, with strong results across many benchmarks at scale.