Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

5 records

0012018Encoders

Peer reviewed

BERT: Pre-training of Deep Bidirectional Transformers

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

BERT pre-trains an encoder by masking random tokens and predicting them from both left and right context, unlike the left-to-right models of the GPT line, plus a next-sentence-prediction objective. The resulting representations condition each token on the full surrounding sentence, and a single added output layer fine-tunes the model for classification, tagging, or span-based question answering. It set new results on GLUE and SQuAD and became the standard encoder for understanding-oriented NLP tasks.

UnknownDifficulty 5/10Verified
Read record Primary source
0022019Encoders

Preprint

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, et al.

RoBERTa is a careful re-examination of how BERT was trained rather than a new model architecture. The authors trained longer on about ten times more text, used bigger batches, generated the masking pattern fresh each time an example is seen (dynamic masking), dropped the next-sentence-prediction task, and trained on full-length sentence sequences. These changes alone pushed performance above the original BERT and matched or beat later models on GLUE, SQuAD, and RACE. The main lesson was methodological: much of what looked like architectural progress was actually the result of undertraining, and controlled comparisons require fixing the training budget.

UnknownDifficulty 4/10Verified
Read record Primary source
0032020Encoders

Peer reviewed

ELECTRA: Pre-training Text Encoders as Discriminators

Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning

Rather than masking tokens and predicting them, ELECTRA uses a small generator to swap some tokens for plausible alternatives, then trains the main model to decide, for every token, whether it was replaced. Because the loss covers all positions rather than the small masked subset, each training step yields more signal per example. This let ELECTRA match or exceed BERT-scale accuracy using substantially less pretraining compute, making strong encoders reachable on smaller budgets.

UnknownDifficulty 5/10Verified
Read record Primary source
0042021Encoders

Peer reviewed

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen

DeBERTa keeps a token's content and its relative position as two distinct vectors and computes attention weights from content-to-content, content-to-position, and position-to-content terms, capturing that a word's relationship to another depends on their distance. Because relative encoding alone loses absolute placement needed for tasks like masked-word prediction, it adds an enhanced mask decoder that folds absolute positions back in just before the output layer. These changes improved sample efficiency and accuracy on language-understanding benchmarks relative to BERT and RoBERTa at comparable model sizes.

UnknownDifficulty 6/10Verified
Read record Primary source
0052019Encoders

Peer reviewed

ALBERT: A Lite BERT

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut

ALBERT decouples the vocabulary embedding size from the hidden size (factorizing that large matrix) and shares the same parameters across all Transformer layers, drastically reducing the model's parameter footprint. It also replaces BERT's next-sentence prediction with a sentence-order prediction task that forces the model to learn inter-sentence coherence rather than topic overlap. Together these let ALBERT scale hidden dimensions and depth without a proportional memory blowup, reaching stronger results than BERT-large at a fraction of the stored parameters.

UnknownDifficulty 5/10Verified
Read record Primary source