Encoder–Decoders · 2020
BART: Denoising Seq2Seq Pre-training
BART pretrains a full encoder-decoder Transformer as a denoising autoencoder that reconstructs original text from arbitrarily corrupted input, unifying BERT-style bidirectional encoding with GPT-style autoregressive generation for both understanding and generation tasks.
Editorial record
Plain-language summary
BART corrupts documents with noise such as token masking, deletion, sentence shuffling, and text-span infilling, then trains a bidirectional encoder plus autoregressive decoder to reconstruct the original. This seq2seq setup means the same pretrained model handles classification (via the encoder) and generation like summarization and translation (via the decoder), rather than needing separate architectures. Text infilling in particular proved effective, and BART reached strong results on generation benchmarks while remaining competitive on discriminative tasks.
Knowledge graph
Relationships
Antecedents
GeneralizesEvidence: Strongly supported
CodeT5 / CodeGen / InCoder / Fill-in-the-Middle
FIM generalizes BART infilling to code
P-342
Descendants
Applies toEvidence: Direct
Attention Is All You Need
Encoder-decoder denoising autoencoder
P-021 architecture
Source record
Provenance
- Record ID
- P-021
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1910.13461
- arXiv:1910.13461
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.