Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

3 records

0012020Encoder–Decoders

Peer reviewed

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

T5 uses an encoder-decoder Transformer trained with a span-corruption denoising objective and represents each task, including ones with numeric or label outputs, as producing a target string from an input string. The paper runs a controlled comparison of architectures, objectives, corpora, and transfer strategies, and pre-trains on the C4 Common Crawl corpus the authors assembled and released. The unified format removed per-task output heads and enabled systematic study of what drives transfer, with strong results across many benchmarks at scale.

UnknownDifficulty 5/10Verified
Read record Primary source
0022020Encoder–Decoders

Peer reviewed

BART: Denoising Seq2Seq Pre-training

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer

BART corrupts documents with noise such as token masking, deletion, sentence shuffling, and text-span infilling, then trains a bidirectional encoder plus autoregressive decoder to reconstruct the original. This seq2seq setup means the same pretrained model handles classification (via the encoder) and generation like summarization and translation (via the decoder), rather than needing separate architectures. Text infilling in particular proved effective, and BART reached strong results on generation benchmarks while remaining competitive on discriminative tasks.

UnknownDifficulty 5/10Verified
Read record Primary source
0032023Encoder–Decoders

Peer reviewed

UL2: Unifying Language Learning Paradigms

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, et al.

UL2 trains on a mixture of denoising objectives (short spans, long spans, and sequential/prefix-LM corruption) and prepends a mode token that tells the model which denoising regime applies, letting it be switched between modes at inference. This bridges the gap where masked-denoising models excelled at fine-tuning while causal LMs excelled at few-shot prompting, giving one recipe that performs on both. The framework is architecture-agnostic and let a single pretrained model be adapted to supervised fine-tuning and prompting without choosing an objective in advance.

UnknownDifficulty 6/10Verified
Read record Primary source