Encoders · 2021
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
DeBERTa introduces disentangled attention, representing each token by separate content and relative-position vectors whose attention is computed across both, plus an enhanced mask decoder that reinjects absolute positions, improving over BERT and RoBERTa at equal or smaller scale.
Editorial record
Plain-language summary
DeBERTa keeps a token's content and its relative position as two distinct vectors and computes attention weights from content-to-content, content-to-position, and position-to-content terms, capturing that a word's relationship to another depends on their distance. Because relative encoding alone loses absolute placement needed for tasks like masked-word prediction, it adds an enhanced mask decoder that folds absolute positions back in just before the output layer. These changes improved sample efficiency and accuracy on language-understanding benchmarks relative to BERT and RoBERTa at comparable model sizes.
Knowledge graph
Relationships
Descendants
ImprovesEvidence: Direct
BERT: Pre-training of Deep Bidirectional Transformers
Disentangled content/position attention
P-024 paper
Depends onEvidence: Strongly supported
Self-Attention with Relative Position Representations / Transformer-XL
Relative position representations underpin disentangled attention
P-024 Sec 2
Source record
Provenance
- Record ID
- P-024
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2006.03654
- arXiv:2006.03654
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.