Encoder–Decoders · 2023
UL2: Unifying Language Learning Paradigms
UL2 introduces Mixture-of-Denoisers, a pretraining scheme that blends several span-corruption and prefix-language-modeling objectives with mode-switching tokens, producing a single model that works across both fine-tuning and in-context few-shot settings regardless of architecture.
Editorial record
Plain-language summary
UL2 trains on a mixture of denoising objectives (short spans, long spans, and sequential/prefix-LM corruption) and prepends a mode token that tells the model which denoising regime applies, letting it be switched between modes at inference. This bridges the gap where masked-denoising models excelled at fine-tuning while causal LMs excelled at few-shot prompting, giving one recipe that performs on both. The framework is architecture-agnostic and let a single pretrained model be adapted to supervised fine-tuning and prompting without choosing an objective in advance.
Knowledge graph
Relationships
Descendants
GeneralizesEvidence: Direct
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
Mixture-of-denoisers generalizes span-corruption
P-022 paper
Source record
Provenance
- Record ID
- P-022
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2205.05131
- arXiv:2205.05131
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.