0011986Neural Foundations
Foundational paper
David Rumelhart, Geoffrey Hinton, Ronald Williams
The paper applies the chain rule to compute, for every weight in a layered network, how much it contributed to the output error, then adjusts weights in the direction that reduces that error. This gave hidden units a way to learn useful internal representations instead of being hand-designed. It made training networks with one or more hidden layers a routine gradient-descent procedure and became the standard method for supervised neural network learning.
UnknownDifficulty 5/10Verified
0022014Neural Foundations
Peer reviewed
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
Instead of forcing the encoder to compress a whole sentence into one vector, this model learns an alignment: for each output word the decoder scores and softly selects relevant source positions to build a context vector. This kept translation quality from degrading as sentences got longer and produced interpretable soft alignments between source and target words. The mechanism became the core building block that the Transformer later generalized.
UnknownDifficulty 5/10Verified
0032015Neural Foundations
Peer reviewed
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
The authors added shortcut connections that add a layer block's input to its output, so each block only has to learn the difference from identity. This made gradients flow through very deep stacks and allowed networks of 50 to 152 layers to train and improve rather than degrade, winning ImageNet 2015. Residual blocks became a default component in most later deep vision and language architectures.
UnknownDifficulty 4/10Verified
0042014Neural Foundations
Peer reviewed
Diederik Kingma, Jimmy Ba
Adam keeps exponentially decaying averages of past gradients and squared gradients, applies bias correction to those estimates, and uses them to scale each parameter's step. This gives well-behaved step sizes across parameters with different gradient magnitudes and works with little tuning across many problems. Its combination of low memory cost and robust default hyperparameters made it the default optimizer for training deep networks.
UnknownDifficulty 4/10Verified
0051936Origins & Computability
Foundational paper
Alan Turing
Turing described a simple imagined device that reads and writes symbols on a tape according to a finite table of rules, and argued this captures anything a human clerk could compute by following steps. Using it he showed that a single 'universal' machine can simulate any other by reading its description, and that the halting problem has no general algorithmic solution. This gave a rigorous definition of 'algorithm' and 'computable', and the universal-machine idea underlies the concept of a programmable general-purpose computer.
UnknownDifficulty 6/10Verified
0061945Origins & Computability
Architecture report
von Neumann
The draft described a design with a central arithmetic unit, a control unit, and a shared memory holding both program and data, communicating over common paths. Because instructions live in modifiable memory, the same hardware can run any program just by loading different contents. This 'stored-program' organization became the template for essentially all subsequent general-purpose computers.
UnknownDifficulty 5/10Verified
0071948Origins & Computability
Foundational paper
Claude Shannon
He measured information in bits using entropy, separating a message's content from its meaning, and modeled communication as a source, channel, and receiver subject to noise. He proved that every channel has a maximum reliable rate (its capacity) and that codes exist to approach it with arbitrarily few errors, and gave limits on lossless compression. These results underpin modern data compression, error-correcting codes, and digital communication and storage.
UnknownDifficulty 6/10Verified
0082017Transformer Architecture
Peer reviewed
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
The paper replaces recurrent and convolutional encoders with stacked multi-head self-attention and feed-forward layers, using positional encodings to retain word order. Because attention relates all positions at once rather than stepping through a sequence, training parallelizes across the sequence and long-range dependencies are captured in a constant number of operations. It set new machine-translation results on WMT English-German and English-French at lower training cost, and became the base architecture for essentially all later large language models.
UnknownDifficulty 6/10Verified
0092020Decoder-Only Language Models
Peer reviewed
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al.
GPT-3 trains a 175-billion-parameter Transformer on a filtered Common Crawl plus other corpora, keeping the next-token objective but scaling roughly 100x over GPT-2. Given a natural-language instruction and a handful of demonstrations in its context window, it performs translation, question answering, arithmetic, and other tasks without weight updates, with accuracy generally rising as more examples are shown. This removed the per-task fine-tuning and labeled-data requirement for many uses and made prompting the primary interface to large models.
UnknownDifficulty 6/10Verified
0102018Encoders
Peer reviewed
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
BERT pre-trains an encoder by masking random tokens and predicting them from both left and right context, unlike the left-to-right models of the GPT line, plus a next-sentence-prediction objective. The resulting representations condition each token on the full surrounding sentence, and a single added output layer fine-tunes the model for classification, tagging, or span-based question answering. It set new results on GLUE and SQuAD and became the standard encoder for understanding-oriented NLP tasks.
UnknownDifficulty 5/10Verified
0112022Scaling Laws & Compute
Peer reviewed
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al.
DeepMind trained over 400 models at varied size and token counts and re-estimated the compute-optimal frontier, finding that for a given compute budget parameters and data should grow together at about a 1:1 ratio rather than favoring size. To demonstrate it they trained Chinchilla, a 70B model on 1.4 trillion tokens, which outperformed the 280B Gopher and other larger models while being cheaper to run at inference. The result redirected the field toward training smaller models on far more data.
UnknownDifficulty 6/10Verified
0122022Alignment & Preference Learning
Peer reviewed
Long Ouyang, Jeff Wu, Xu Jiang, et al.
Labelers wrote demonstrations to supervise-fine-tune GPT-3, then ranked model outputs to train a reward model, and PPO optimized the policy against that reward. Outputs from the 1.3B InstructGPT model were preferred to the 175B GPT-3's despite being about 100x smaller, with gains in truthfulness and reductions in toxic generation. The three-stage SFT-then-reward-model-then-PPO recipe became the standard alignment pipeline behind instruction-following chat models.
UnknownDifficulty 6/10Verified
0132021Multimodality
Peer reviewed
Alec Radford, Jong Wook Kim, Chris Hallacy, et al.
Two encoders (one for images, one for text) are trained so that a photo and its matching caption land close together in a shared embedding space while mismatched pairs are pushed apart. To classify a new image with no task-specific training, you write candidate labels as text prompts and pick the one whose embedding is closest to the image. Because the labels are open-ended text rather than a fixed category list, one model transfers to many recognition tasks and became a reusable component for retrieval and for later image-generation and vision-language systems.
UnknownDifficulty 5/10Verified
0142014Neural Foundations
Peer reviewed
Ilya Sutskever, Oriol Vinyals, Quoc Le
The paper used one multilayer LSTM to read an entire source sentence into a single vector and a second LSTM to generate the target sentence from that vector. A key practical trick was reversing the order of source words, which shortened the average distance between corresponding input and output tokens and made optimization easier, pushing WMT'14 English-to-French BLEU to competitive levels. It showed that a purely neural, general-purpose sequence model could rival phrase-based statistical MT and became the template for later encoder-decoder systems.
UnknownDifficulty 5/10Verified
0152016Neural Foundations
Preprint
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey Hinton
The method computes each layer's normalization statistics over the units within one example, so behavior is identical during training and testing and does not depend on batch composition. This made it effective for RNNs and sequence models where batch normalization worked poorly, stabilizing and speeding up training. Layer normalization became standard in Transformer architectures.
UnknownDifficulty 4/10Verified
0162015Neural Foundations
Peer reviewed
Rico Sennrich, Barry Haddow, Alexandra Birch
BPE starts from characters and iteratively merges the most frequent adjacent symbol pairs to build a vocabulary of subword units, so any word can be encoded as a sequence of known pieces. This let translation models handle rare words, compounds, and morphology without a huge word vocabulary or a separate back-off mechanism. Subword segmentation via BPE became standard tokenization for most later NMT and language models.
UnknownDifficulty 4/10Verified
0172012Neural Foundations
Peer reviewed
Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton
AlexNet is a deep convolutional network with roughly 60 million parameters trained on over a million labeled images across 1,000 categories, using ReLU activations for faster training, dropout to limit overfitting, data augmentation, and two GPUs to fit the model in memory. Combining these let it cut the ImageNet top-5 error far below the previous best. Its result shifted computer vision toward deep learning and drove wide adoption of GPU-trained convolutional networks.
UnknownDifficulty 4/10Verified
0181997Neural Foundations
Peer reviewed
Sepp Hochreiter, Jürgen Schmidhuber, Felix Gers
Each LSTM unit keeps a memory cell whose value is protected by gates that control when new information is written in, retained, or read out, so the cell can hold information across many time steps. Because the cell passes its state forward with near-constant behavior when the gates allow, gradients do not shrink away over long sequences the way they do in plain recurrent networks. This let recurrent networks learn dependencies spanning hundreds of steps and made them effective for sequence tasks such as speech and language modeling.
UnknownDifficulty 5/10Verified
0192009Neural Foundations
Peer reviewed
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei
ImageNet organised millions of human-labelled images into thousands of categories following the WordNet hierarchy, assembled through large-scale crowdsourcing. It supplied the data and the annual ILSVRC challenge that made it possible to train and fairly compare very large vision models. AlexNets 2012 win on this benchmark is widely taken as the moment deep learning became the dominant approach in vision.
UnknownDifficulty 3/10Verified
0201937Origins & Computability
Thesis
Claude Shannon
In his master's thesis Shannon mapped open and closed switches onto Boolean true/false values and series/parallel wiring onto logical operations. This meant a designer could write a switching circuit as a logic equation, reduce it algebraically to use fewer components, and verify it behaves correctly. The result became the standard method for designing the digital logic inside telephone systems and later computers.
UnknownDifficulty 5/10Verified
0211943Origins & Computability
Foundational paper
Warren McCulloch, Walter Pitts
They abstracted a neuron as a device that sums weighted inputs and fires if the total crosses a threshold, ignoring biological detail. They proved that networks of such idealized neurons can implement any logical proposition, linking brain-style computation to formal logic and Turing's model. This established the idea that networks of simple connected units can compute, providing the conceptual starting point for later artificial neural networks.
UnknownDifficulty 5/10Verified
0221945Origins & Computability
Vision essay
Vannevar Bush
Bush described a hypothetical machine that would hold a person's books, records, and notes on microfilm and let the user create named 'trails' of associative links between items. He argued that organizing knowledge by association rather than rigid indexing would match how people actually think and recall. The essay is widely credited as an early inspiration for hypertext, personal information systems, and the linked structure of the web.
UnknownDifficulty 4/10Verified
0231950Origins & Computability
Foundational paper
Alan Turing
Turing proposed the 'imitation game', in which a judge exchanges typed messages with a hidden human and a hidden machine and tries to tell which is which. He suggested that a machine passing this test should count as intelligent for practical purposes, sidestepping arguments over the definition of thought. The paper also anticipated and answered common objections and outlined machine learning, setting an early behavioral benchmark and framing debates in artificial intelligence.
UnknownDifficulty 4/10Verified
0241945Origins & Computability
Institutional milestone
J. Robert Oppenheimer, Leslie Groves, John von Neumann, et al.
Between 1942 and 1945 the United States gathered physicists, chemists, and engineers at Los Alamos and other sites to build an atomic weapon, coordinating theory, experiment, and mass manufacturing on an unprecedented scale. The effort showed that a well-funded, centrally managed team could compress decades of research into a few years. It created the postwar model of the national laboratory and the pattern of governments financing ambitious technical projects. Many figures and methods from this work, including early electronic computation, carried directly into the founding of modern computer science.
UnknownDifficulty 3/10Verified
0251949Origins & Computability
Method
Stanislaw Ulam, John von Neumann, Nicholas Metropolis
Working on nuclear physics problems at Los Alamos, Stanislaw Ulam, John von Neumann, Nicholas Metropolis, and colleagues realized that random sampling on a computer could approximate answers to equations too complex to solve directly. The 1953 Metropolis algorithm added a rule for sampling from a probability distribution by accepting or rejecting proposed moves, making it possible to simulate systems in equilibrium. These techniques became standard tools across physics, statistics, and later machine learning. Much of modern probabilistic modeling and Bayesian computation traces back to this sampling idea.
UnknownDifficulty 6/10Verified
0261956Origins & Computability
Founding event
John McCarthy, Marvin Minsky, Claude Shannon, Nathaniel Rochester
In the summer of 1956, John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon organized a workshop at Dartmouth College to study whether machines could be made to reason, learn, and use language. Their proposal coined the term 'artificial intelligence' and argued that every aspect of intelligence could in principle be specified well enough to be automated. The meeting gathered the researchers who would lead the field for decades. It set the research directions, from problem solving to language and learning, that defined AI's first generation.
UnknownDifficulty 3/10Verified
0271958Origins & Computability
Foundational paper
Frank Rosenblatt
In 1958 Frank Rosenblatt described the perceptron, a simple network that combines weighted inputs and produces a decision, and a procedure that updates those weights whenever it makes a mistake. This gave a working example of a machine that improves its performance by seeing labeled data rather than being explicitly programmed. The perceptron could learn to separate patterns that are linearly separable, though later work showed its limits on harder problems. Its learning rule and layered structure are the foundation of today's deep learning.
UnknownDifficulty 5/10Verified
0282021Transformer Architecture
Peer reviewed
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu
RoPE multiplies each pair of embedding dimensions by a position-dependent rotation before computing attention, so a token's position is baked into the query/key geometry rather than added as a separate vector. Because two rotated vectors' dot product is a function of their offset, attention scores naturally become relative-position aware. This gives cleaner extrapolation to sequence lengths not seen in training and is now the default position scheme in most open large language models.
UnknownDifficulty 6/10Verified
0292018Decoder-Only Language Models
Technical report
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever
GPT-1 pre-trains a left-to-right Transformer on a large corpus of books to predict the next token, then adapts to each downstream task with a small supervised fine-tuning step and task-specific input formatting. This two-stage recipe removed the need to hand-design a separate model per task and to rely on scarce labeled data. It improved results on entailment, question answering, semantic similarity, and classification, establishing generative pre-training plus fine-tuning as a general NLP method.
UnknownDifficulty 5/10Verified
0302019Decoder-Only Language Models
Technical report
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever
GPT-2 trains a 1.5B-parameter Transformer on WebText, a large corpus scraped from outbound Reddit links, using the same next-token objective as GPT-1. The paper shows the model handles reading comprehension, translation, summarization, and question answering in a zero-shot setting when the task is phrased as text, without any task-specific training. This reframed NLP tasks as special cases of language modeling and gave early evidence that capability scales with model and data size.
UnknownDifficulty 5/10Verified
0312020Encoder–Decoders
Peer reviewed
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
T5 uses an encoder-decoder Transformer trained with a span-corruption denoising objective and represents each task, including ones with numeric or label outputs, as producing a target string from an input string. The paper runs a controlled comparison of architectures, objectives, corpora, and transfer strategies, and pre-trains on the C4 Common Crawl corpus the authors assembled and released. The unified format removed per-task output heads and enabled systematic study of what drives transfer, with strong results across many benchmarks at scale.
UnknownDifficulty 5/10Verified
0322020Scaling Laws & Compute
Preprint
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al.
The authors trained many Transformer language models across orders of magnitude in parameters, data, and compute, then fit power laws to the loss curves. They showed loss scales predictably with each factor when the others are not bottlenecked, and that within their observed range larger models are more sample-efficient, so given fixed compute it was better to train very large models on comparatively less data and stop early. This gave labs a quantitative basis to forecast returns from more compute and allocate budget before committing to a run.
UnknownDifficulty 6/10Verified
0332017Mixture-of-Experts
Peer reviewed
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean
The layer holds up to thousands of expert feed-forward networks and a gating function that selects only the top-k experts per token, so only a small fraction of parameters activate per example. The authors add noisy top-k gating and load-balancing losses to keep experts from collapsing onto a few favorites, and apply it to stacked LSTM language and translation models. This showed conditional computation could scale parameter count by orders of magnitude on real tasks, and it is the direct ancestor of transformer MoE models.
UnknownDifficulty 6/10Verified
0342022Mixture-of-Experts
Peer reviewed
William Fedus, Barret Zoph, Noam Shazeer
Switch replaces the top-k MoE gate with top-1 routing so each token goes to exactly one expert, which cuts routing computation and communication while keeping a fixed compute budget per token. The paper adds selective precision, capacity factors and an auxiliary load-balancing loss, and expert-parallel sharding to make this stable at scale. The result trains far faster than dense baselines at equal FLOPs and reaches trillion-parameter counts, giving a practical template for sparse scaling.
UnknownDifficulty 6/10Verified
0352019Distributed Training
Infrastructure paper
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al.
NVIDIA partitioned the Transformer's attention and MLP weight matrices column- and row-wise across GPUs, arranging the splits so only two all-reduce operations per layer are needed in the forward and backward passes, implemented directly in PyTorch. This let them train models up to 8.3 billion parameters and scale efficiently to 512 GPUs. The approach became a standard building block, later combined with data and pipeline parallelism, for training multi-billion-parameter models.
UnknownDifficulty 6/10Verified
0362020Distributed Training
Infrastructure paper
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
The authors observed that standard data parallelism stores a full copy of optimizer states, gradients, and weights on every GPU, wasting memory as models grow. ZeRO shards these three components across the data-parallel group in three progressive stages, reconstructing pieces via communication only when needed, so per-device memory falls roughly in proportion to the number of devices while keeping data-parallel efficiency. Implemented in DeepSpeed, it enabled training of models into the hundreds of billions of parameters on existing clusters.
UnknownDifficulty 6/10Verified
0372022Alignment & Preference Learning
Peer reviewed
Jason Wei, Maarten Bosma, Quoc V. Le, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, Daniel Khashabi
The authors took a pretrained model and fine-tuned it on many existing NLP datasets that were each rewritten as instructions (for example, 'Is this review positive or negative?'). After this instruction tuning, the model could handle new kinds of tasks it had not seen during training, just from reading the instruction. This established instruction tuning as a general method for making base models usable without few-shot prompting, and improved zero-shot performance across a range of benchmarks.
UnknownDifficulty 4/10Verified
0382022Alignment & Preference Learning
Technical report
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al.
In a supervised phase the model critiques and revises its own responses against a short list of natural-language principles (a 'constitution'), and in an RL phase a model rather than humans ranks response pairs for harmlessness to train the preference model (RLAIF). This let the assistant refuse or push back on harmful requests while explaining its reasoning instead of giving evasive non-answers. It cut human labeling of toxic content and made the value targets explicit and editable as written rules.
UnknownDifficulty 6/10Verified
0392023Alignment & Preference Learning
Peer reviewed
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, Chelsea Finn
Direct Preference Optimization uses the mathematical link between reward and optimal policy to rewrite preference learning so the language model itself is optimized directly on chosen-versus-rejected examples with a simple supervised loss. There is no reward model to train and no PPO sampling, so it is more stable and cheaper while matching or beating PPO-based RLHF on preference tuning. This made preference alignment practical for teams without reinforcement-learning infrastructure.
UnknownDifficulty 6/10Verified
0402022Reasoning & Test-Time Compute
Peer reviewed
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou
By putting a few exemplars that spell out step-by-step worked solutions into the prompt, the model imitates that format and reasons through arithmetic, commonsense, and symbolic problems one step at a time. The benefit appears mainly at large model scale and substantially raised accuracy on benchmarks like GSM8K math word problems. It made intermediate-computation prompting a standard, training-free way to get harder reasoning out of existing models.
UnknownDifficulty 5/10Verified
0412025Reasoning & Test-Time Compute
Technical report
DeepSeek-AI, Daya Guo, Zhihong Shao, Wenfeng Liang
The team applied reinforcement learning where the reward comes from verifying the final result (a math answer being correct or code passing tests) rather than from human preference labels. Under this pressure the model learned on its own to write longer reasoning, check its work, and backtrack, with a pure-RL variant (R1-Zero) developing these behaviors from a base model before a small amount of supervised data was added for readability. This demonstrated that reinforcement learning with verifiable rewards (RLVR) is a scalable path to reasoning models, and the released weights made such training widely reproducible.
UnknownDifficulty 7/10Verified
0422023Agents & Tool Use
Peer reviewed
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao
The method prompts a language model to alternate between writing a reasoning step and issuing an action, such as a query to a search API, then feeding the observation back before the next thought. This lets the model plan, look things up to correct itself, and reduce fabricated facts on knowledge tasks like HotpotQA and FEVER, while the reasoning traces make its behavior on interactive benchmarks more legible. It became a common template for tool-using and agentic prompting because it needs no fine-tuning, only few-shot exemplars.
UnknownDifficulty 5/10Verified
0432021Multimodality
Peer reviewed
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al.
The model splits an image into a grid of small patches (e.g. 16x16 pixels), linearly embeds each patch as if it were a word token, adds position embeddings, and feeds the sequence into a standard Transformer encoder. With no convolutions, it learns spatial structure from data alone, which works well only when pre-trained on very large image datasets. This let the same architecture used for language be reused for vision and scaled up with more data and compute rather than hand-designed vision components.
UnknownDifficulty 5/10Verified
0442020Retrieval & Memory
Peer reviewed
Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al.
RAG pairs a trained generator with a retriever that fetches relevant passages from an external corpus (a Wikipedia index searched with dense vectors), then conditions the output on those passages. Both the retriever's query encoder and the generator are trained together, and retrieved documents can be swapped or updated without retraining the model. This lets a model cite and use up-to-date or domain-specific knowledge, reducing fabricated answers on knowledge-intensive tasks like open-domain question answering.
UnknownDifficulty 5/10Verified
0452023Open-Weight Model Families
Technical report
Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al.
Meta trained a family from 7B to 65B parameters exclusively on public datasets, pushing token counts well past Chinchilla-optimal points to favor cheaper inference. The 13B model matched or beat GPT-3 (175B) on many benchmarks and the 65B was competitive with Chinchilla and PaLM. Its release under a research license, and the subsequent weight availability, seeded a large open ecosystem of fine-tuned derivatives.
Open weightDifficulty 5/10Verified
0462024Open-Weight Model Families
Technical report
Hugo Touvron, Abhimanyu Dubey, et al.
The paper describes how the Llama 3 models (up to 405B parameters) were built: a large curated pretraining corpus, a standard dense transformer architecture scaled up, and a post-training pipeline of supervised fine-tuning and preference optimization for instruction following. It also covers extended context length, multilingual and code capabilities, and safety tuning. Because the weights are downloadable under a community license, it gave researchers and companies a strong base model to run and fine-tune locally, though the license and undisclosed training data mean it is open-weight rather than fully open source.
Open weightDifficulty 5/10Verified
0472024Open-Weight Model Families
Technical report
DeepSeek-AI, Damai Dai, Aixin Liu, Wenfeng Liang
DeepSeek-V3 is a mixture-of-experts model where only a small fraction of its total parameters activate per token, keeping compute per token low. It introduces multi-head latent attention, which compresses the attention key/value cache into a smaller latent representation to reduce memory during inference, and pairs it with an auxiliary-loss-free scheme for balancing which experts get used. Trained in FP8 mixed precision, it reached quality comparable to leading models while using far fewer GPU-hours, and the weights are downloadable, making it open-weight.
Open weightDifficulty 6/10Verified
0482022Inference & Serving
Peer reviewed
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, Jay Shah
The kernel splits queries, keys, and values into blocks that fit in fast on-chip SRAM, computes softmax incrementally with running statistics, and recomputes intermediates during the backward pass instead of storing them, so memory scales linearly rather than quadratically in sequence length. This yields wall-clock speedups and lower memory use without approximation, giving identical results to standard attention. It enabled training and serving with longer context windows and became a default attention implementation in mainstream frameworks.
UnknownDifficulty 6/10Verified
0492023Inference & Serving
Peer reviewed
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Ion Stoica
The KV cache for each sequence is stored in small blocks tracked by a lookup table, so memory is allocated on demand and blocks can be shared across sequences that share a prefix, with copy-on-write on divergence. This cuts memory waste to a few percent and raises the number of requests that can be batched together, and it is the core of the vLLM serving engine. The result was substantially higher serving throughput at the same latency compared to prior systems.
UnknownDifficulty 5/10Verified
0502022Inference & Serving
Peer reviewed
Edward J. Hu, Yelong Shen, Weizhu Chen, Tim Dettmers, Luke Zettlemoyer
Instead of updating a weight matrix directly, the method learns a low-rank product added to it, so only a tiny fraction of parameters are trained and the small adapters can be stored per task and swapped in. Because the update can be merged back into the base weights after training, there is no extra latency at inference, unlike adapter layers. On GPT-3 175B it reduced trainable parameters by roughly ten-thousand-fold while matching full fine-tuning quality, making per-task customization of large models cheap and modular.
UnknownDifficulty 5/10Verified
0512024Evaluation & Benchmarks
Benchmark paper
Carlos E. Jimenez, John Yang, et al.
The benchmark draws over 2,000 issue-and-pull-request pairs from popular Python repositories, giving the model an issue description and the codebase and asking it to produce a patch, then judging it by running the maintainers' fail-to-pass and pass-to-pass tests. Success demands navigating a large codebase, editing multiple files, and understanding project context, and at release even strong models solved only a low single-digit percentage. It became a leading measure of practical software-engineering ability and drove work on agentic coding systems.
UnknownDifficulty 5/10Verified
0522019Scaling Laws & Compute
Essay
Richard Sutton
Sutton observes that in game-playing, speech, and vision, hand-crafted domain knowledge gave early wins but was eventually overtaken by search and learning methods that scale with compute. The uncomfortable conclusion is that human insight matters less than the ability to exploit growing computation. The essay became a rallying text for scaling-first research.
UnknownDifficulty 3/10Verified
0532021Multimodality
Peer reviewed
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Demis Hassabis
AlphaFold pairs an attention-based Evoformer that reasons jointly over multiple-sequence alignments and residue-pair representations with a structure module that directly predicts atomic coordinates, trained end-to-end with protein geometry built in. At the CASP14 assessment it predicted many structures to within experimental error, a step change over prior methods. DeepMind released the code and a public database of predicted structures, reshaping computational structural biology.
UnknownDifficulty 7/10Verified
0542015Inference & Serving
Preprint
Geoffrey Hinton, Oriol Vinyals, Jeff Dean
Instead of training only on hard labels, the student is trained on the teacher’s full probability distribution softened by a temperature, so it inherits the relative confidences the teacher assigns across classes — information the hard label throws away. Students recover much of the teacher’s performance at a fraction of the parameters and compute. Distillation became a standard compression tool and, later, the way strong small language models are trained from frontier teachers.
UnknownDifficulty 5/10Verified
0552003Neural Foundations
Peer reviewed
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin
Instead of treating words as discrete symbols, the model maps each word to a real-valued vector and feeds the vectors of the preceding words into a neural network that outputs a probability distribution over the next word. Words used in similar contexts end up with similar vectors, so the model can assign reasonable probability to word sequences it never saw in training. This showed that a single network could learn word features and the probability function together, and set the template later used by neural word embeddings and language models.
UnknownDifficulty 5/10Verified
0562013Neural Foundations
Preprint
Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean
The models drop the hidden nonlinear layers of earlier neural language models and instead train a shallow network to predict a word from its surrounding context (CBOW) or the surrounding context from a word (skip-gram). This makes training fast enough to run on billions of words, producing vectors where semantic and syntactic relationships appear as consistent offsets (the vector arithmetic 'king minus man plus woman' lands near 'queen'). It made high-quality word embeddings cheap to compute and widely reusable as input features for other NLP systems.
UnknownDifficulty 4/10Verified
0572014Neural Foundations
Peer reviewed
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov
During training, each unit is kept with some probability and dropped otherwise, so the network cannot rely on any specific combination of units and must learn features that are useful on their own. At test time all units are used with their outputs scaled to match, which approximates averaging over the many thinned networks seen during training. This reduced overfitting on a range of vision and speech tasks and became a standard technique for training large networks with limited data.
UnknownDifficulty 3/10Verified
0582010Neural Foundations
Peer reviewed
Xavier Glorot, Yoshua Bengio, Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
The paper measures how the variance of activations and back-propagated gradients changes from layer to layer under different setups, showing that poor scaling makes deep networks with saturating activations (like sigmoid) hard to train. It proposes initializing weights with a variance set from the number of incoming and outgoing connections so that signals neither vanish nor blow up as they pass through the layers. This initialization let deeper networks converge more reliably and became a default starting point for many architectures.
UnknownDifficulty 5/10Verified
0592010Neural Foundations
Peer reviewed
Vinod Nair, Geoffrey Hinton, Xavier Glorot, Antoine Bordes, Yoshua Bengio
A ReLU passes positive inputs through unchanged and outputs zero for negative inputs, so its gradient is a constant one wherever the unit is active instead of shrinking toward zero the way sigmoid and tanh gradients do. This keeps gradients flowing through many layers and produces sparse activations that are cheap to compute. Using ReLUs let deep networks train faster and reach lower error, and it became the default activation for most feedforward and convolutional networks.
UnknownDifficulty 3/10Verified
0602015Neural Foundations
Peer reviewed
Sergey Ioffe, Christian Szegedy
For each mini-batch, the method normalizes a layer's pre-activation values to zero mean and unit variance, then applies a learned scale and shift so the layer can still represent what it needs. This keeps the distribution of inputs to each layer more stable as the weights below it change, which allows much higher learning rates and less sensitivity to initialization. It sped up training substantially, added a mild regularizing effect, and became a standard component of deep convolutional networks.
UnknownDifficulty 4/10Verified
0611998Neural Foundations
Peer reviewed
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner
The network stacks convolutional layers that apply the same small learned filters across the image with pooling layers that reduce resolution, so it detects local patterns regardless of position while keeping the parameter count low through weight sharing. Trained on labeled digit images, it learns its own feature detectors instead of relying on manually designed ones. It achieved strong accuracy on handwritten digit recognition and was deployed to read amounts on bank checks, demonstrating a practical gradient-trained vision system.
UnknownDifficulty 4/10Verified
0622006Neural Foundations
Peer reviewed
Geoffrey Hinton, Simon Osindero, Yee-Whye Teh, Ruslan Salakhutdinov
The method builds a deep belief network by training each layer in turn to model the activations of the layer below, giving the full network a sensible starting point before ordinary gradient fine-tuning. Applied to an autoencoder, this two-stage approach learns a compact code that reconstructs the input better than linear methods like PCA. The work provided a practical way to train deep networks before better initialization and activation functions existed, and helped renew interest in deep learning.
UnknownDifficulty 5/10Verified
0631988Neural Foundations
Peer reviewed
Richard S. Sutton
Sutton showed how to learn multi-step predictions incrementally: instead of comparing a prediction only to the eventual outcome, TD updates each prediction using the difference between successive predictions along the way. This let learning happen online, step by step, and made better use of experience than earlier supervised methods. TD became the computational core of most later reinforcement learning, including value estimation in games and control.
UnknownDifficulty 6/10Verified
0641992Neural Foundations
Peer reviewed
Christopher J. C. H. Watkins, Peter Dayan
Watkins and Dayan defined a rule that updates an estimate of the long-run reward for each state-action pair (the Q-value) from observed transitions and rewards. Because updates use the best available next action rather than the action actually taken, the agent can learn the optimal policy while still exploring. They gave a convergence proof, giving reinforcement learning a model-free method with mathematical guarantees that underpins much later work.
UnknownDifficulty 5/10Verified
0651992Neural Foundations
Peer reviewed
Ronald J. Williams
Williams derived how to nudge the weights of a stochastic network so that actions leading to higher reward become more likely, using only the reward signal and the probability of the chosen action. The estimate of the gradient is unbiased and needs no model of the environment. This policy-gradient approach became the basis for later actor-critic and deep policy-optimization methods used to train agents and language models.
UnknownDifficulty 6/10Verified
0662015Neural Foundations
Peer reviewed
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al.
Mnih and colleagues trained a network to estimate action values from screen images, using two stabilizing tricks: an experience replay buffer that reuses and decorrelates past transitions, and a periodically updated target network. The same model and hyperparameters learned to play 49 Atari games from pixels and reward alone. It showed that deep networks could serve as reliable function approximators in reinforcement learning, launching the deep RL field.
UnknownDifficulty 6/10Verified
0672016Neural Foundations
Peer reviewed
David Silver, Aja Huang, Chris J. Maddison, Julian Schrittwieser, Thomas Hubert, Demis Hassabis
AlphaGo paired a policy network (which moves to consider) and a value network (who is winning) with tree search to defeat top human Go players. AlphaZero simplified this into a single network trained purely by playing against itself, guided by search, and generalized to chess and shogi. The work showed that self-play plus search could reach superhuman strength without human examples or handcrafted evaluation.
UnknownDifficulty 6/10Verified
0682013Neural Foundations
Peer reviewed
Diederik P. Kingma, Max Welling
Kingma and Welling framed generation as learning a latent variable model and approximating its intractable posterior with a neural encoder. Their key move, the reparameterization trick, rewrote random sampling so gradients could flow through it, allowing the encoder and decoder to be trained together by ordinary backpropagation on a single objective (the evidence lower bound). This gave a scalable way to learn continuous latent representations and generate new data, widely used for images and representation learning.
UnknownDifficulty 6/10Verified
0692014Neural Foundations
Peer reviewed
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al.
Goodfellow and colleagues set up a game between two networks: a generator that turns noise into samples and a discriminator that tries to tell real data from generated data. Training the two in competition drives the generator to match the real data distribution without needing an explicit likelihood. This adversarial framework produced sharp, realistic image generation and spawned a large family of follow-on models for synthesis and translation.
UnknownDifficulty 6/10Verified
0701854Origins & Computability
Foundational treatise
George Boole
Boole treated statements as variables taking values that behave like 0 and 1, with operations for 'and', 'or', and 'not' following consistent algebraic rules. This turned deductive reasoning into symbol manipulation that could be checked mechanically rather than by intuition. The system later became the mathematical basis for digital logic circuits and for how computers represent and evaluate logical conditions.
UnknownDifficulty 4/10Verified
0711843Origins & Computability
Foundational design
Charles Babbage, Ada Lovelace
The Analytical Engine was a proposed mechanical calculator with a separate store and processing 'mill', able to follow instructions and branch conditionally rather than compute one fixed formula. In her annotations to a paper describing it, Lovelace wrote out a step-by-step procedure for computing Bernoulli numbers, often cited as the first published algorithm intended for a machine. She also argued the engine could operate on symbols beyond numbers, anticipating general-purpose computation, though the machine was never built.
UnknownDifficulty 4/10Verified
0721948Origins & Computability
Foundational book
Norbert Wiener
Cybernetics framed systems as maintaining goals by sensing their output and feeding it back to adjust their input, whether the system is a thermostat, an animal, or a servomechanism. Wiener drew together control theory, feedback, and communication to treat purpose and self-correction as engineering problems. The framework influenced control engineering, early thinking about learning machines, and how researchers modeled adaptive and self-regulating behavior.
UnknownDifficulty 5/10Verified
0731949Origins & Computability
Foundational book
Donald Hebb
Hebb argued that when one neuron persistently helps fire another, the link between them grows stronger, often summarized as 'cells that fire together wire together'. He used this to explain how groups of neurons form 'cell assemblies' that represent learned concepts and associations. The rule gave neuroscience a concrete account of learning and became a foundational principle for training weights in artificial neural networks.
UnknownDifficulty 4/10Verified
0741943Origins & Computability
Wartime engineering
Alan Turing, Gordon Welchman, Tommy Flowers, Bill Tutte
During the Second World War, British codebreakers built machines to break the German Enigma and Lorenz ciphers by mechanizing the search through vast numbers of possible settings. The Bombe automated the logical elimination of impossible key configurations, while Colossus used electronic valves to process teleprinter traffic at high speed. This work demonstrated that computation could replace human labor on structured reasoning tasks. It also produced engineering experience and people, including Alan Turing, who shaped the first general-purpose computers.
UnknownDifficulty 5/10Verified
0751944Origins & Computability
Foundational book
John von Neumann, Oskar Morgenstern
In 1944 John von Neumann and Oskar Morgenstern set out a mathematical theory of how rational agents should act when their outcomes depend on each other's choices. They formalized games, strategies, and payoffs, and proved results such as the minimax theorem for zero-sum games, along with a theory of utility for decisions under uncertainty. The book gave economics and later computer science a rigorous way to reason about competition and cooperation. Its concepts underpin work on reinforcement learning, mechanism design, and multi-agent AI.
UnknownDifficulty 6/10Verified
0761964Origins & Computability
Foundational paper
Ray Solomonoff, Andrey Kolmogorov, Gregory Chaitin
In the 1960s Ray Solomonoff, Andrey Kolmogorov, and Gregory Chaitin independently proposed measuring how complex a string of data is by the size of the smallest computer program that can generate it. A string that needs a long program is effectively random, while a compressible one is simple. Solomonoff used this idea to build a theory of prediction that favors the simplest explanation consistent with the data. These concepts connect compression, probability, and learning, and they inform how researchers reason about generalization and Occam's razor in machine learning.
UnknownDifficulty 7/10Verified
0772019Transformer Architecture
Peer reviewed
Biao Zhang, Rico Sennrich
RMSNorm rescales each activation vector by its root-mean-square magnitude and applies a learned gain, skipping the mean-subtraction step that standard LayerNorm performs. The authors argue the re-centering in LayerNorm contributes little and that re-scaling is what actually stabilizes training. The result is fewer operations and less compute per layer at comparable accuracy, which is why it replaced LayerNorm in LLaMA and many later transformer stacks.
UnknownDifficulty 4/10Verified
0782020Transformer Architecture
Preprint
Noam Shazeer
The paper swaps the standard two-matrix ReLU feed-forward block for GLU variants, where one linear projection is elementwise-multiplied by a gated (sigmoid, GELU, or Swish) version of another projection. In controlled pretraining and fine-tuning comparisons, these gated blocks lower perplexity and improve downstream scores over the plain ReLU feed-forward. SwiGLU became the standard feed-forward design in PaLM, LLaMA, and most subsequent large models.
UnknownDifficulty 4/10Verified
0792020Transformer Architecture
Peer reviewed
Ruibin Xiong, Yunchang Yang, Di He, et al.
The paper studies where the layer-normalization step sits relative to the residual connection in a Transformer block. In the original Post-LN design, gradients near the output layer are large at initialization, so training diverges unless you start with a small learning rate and slowly warm it up. By moving the normalization to the input of each sub-layer (Pre-LN), gradients become bounded and roughly uniform across depth, so models train stably without warm-up, tolerate larger learning rates, and converge faster. This made deep Transformers easier and cheaper to train and is why most later large models adopt the Pre-LN arrangement.
UnknownDifficulty 5/10Verified
0802019Transformer Architecture
Preprint
Noam Shazeer
In standard multi-head attention each head has its own key and value projections, so the per-step key/value cache that dominates incremental decoding is large and bandwidth-bound. MQA keeps multiple query heads but collapses to one shared key/value head, shrinking that cache by the number of heads and making token-by-token generation much faster. The trade-off is a small quality drop, but the decoding speedup made MQA a common choice for inference-heavy models and set up the later GQA compromise.
UnknownDifficulty 5/10Verified
0812023Transformer Architecture
Peer reviewed
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai
GQA divides query heads into groups that each share one key/value head, giving a tunable point between full multi-head attention (best quality, large cache) and multi-query attention (smallest cache, some quality loss). The authors also show you can convert an already-trained multi-head model to GQA by mean-pooling its key/value heads and briefly continuing training. This recovers most of MQA's inference speedup while keeping quality close to multi-head, and it became the standard attention layout in LLaMA 2/3-scale models.
UnknownDifficulty 5/10Verified
0822019Encoders
Preprint
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, et al.
RoBERTa is a careful re-examination of how BERT was trained rather than a new model architecture. The authors trained longer on about ten times more text, used bigger batches, generated the masking pattern fresh each time an example is seen (dynamic masking), dropped the next-sentence-prediction task, and trained on full-length sentence sequences. These changes alone pushed performance above the original BERT and matched or beat later models on GLUE, SQuAD, and RACE. The main lesson was methodological: much of what looked like architectural progress was actually the result of undertraining, and controlled comparisons require fixing the training budget.
UnknownDifficulty 4/10Verified
0832021Mixture-of-Experts
Peer reviewed
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, et al.
GShard scales Transformers by replacing some dense feed-forward layers with a large set of expert sub-networks, where a learned gating function routes each token to only a couple of experts, so total parameters grow without a proportional growth in compute per token. The other half of the contribution is a lightweight annotation API that lets the programmer mark how tensors should be partitioned, after which the compiler automatically shards the computation and inserts the needed cross-device communication. Together these let the authors train a 600-billion-parameter multilingual translation model across 2048 TPU cores in a few days. The work made conditional computation and large-scale model parallelism practical without rewriting model code per device layout.
UnknownDifficulty 6/10Verified
0842024Mixture-of-Experts
Technical report
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, et al.
Mixtral 8x7B is a decoder-only Transformer where each feed-forward layer is replaced by eight expert networks and a small router that selects two experts for every token. Because only two of eight experts run per token, the model holds roughly 47 billion parameters in memory but does the compute of a roughly 13-billion-parameter model at inference. Released under an open license, it matched or exceeded much larger dense models such as Llama 2 70B on most benchmarks while being faster to serve. It demonstrated that sparse expert routing could deliver a strong, openly available model with a favorable quality-to-active-compute ratio.
UnknownDifficulty 5/10Verified
0852018Data, Corpora & Tokenization
Software release
Taku Kudo, John Richardson
SentencePiece is a tokenizer that learns a subword vocabulary straight from raw text, without needing a separate word-splitting step that most earlier pipelines assumed. It escapes whitespace as a normal symbol (the underscore marker) so that tokenizing and detokenizing are exactly reversible, which matters for languages like Japanese or Chinese that do not put spaces between words. It supports both byte-pair-encoding and unigram-language-model segmentation and ships as a library with fixed, serializable models so the same text always maps to the same tokens. This standardized, language-agnostic tokenization step is now a default component in many multilingual and non-English NLP systems.
Source availableDifficulty 4/10Verified
0862022Data, Corpora & Tokenization
Peer reviewed
Katherine Lee, Daphne Ippolito, Andrew Nystrom, et al.
The authors measured duplication in common pretraining datasets and found that many sequences appear many times, including substantial overlap between training and evaluation sets. They built two deduplication tools: an exact-substring method using suffix arrays to find long repeated spans, and an approximate document-level method using MinHash to catch near-duplicates. Training on the deduplicated data reduced how often models regurgitated memorized text verbatim, gave more trustworthy evaluation numbers by removing leaked test examples, and required fewer training steps to reach a given accuracy. The paper made corpus deduplication a standard preprocessing step for language-model training.
UnknownDifficulty 4/10Verified
0872021Data, Corpora & Tokenization
Dataset paper
Leo Gao, Stella Biderman, Sid Black, et al.
EleutherAI combined 22 sources spanning academic writing, code, books, legal and other domains into a single 800GB corpus, deliberately weighting high-quality and specialized text rather than relying only on filtered Common Crawl. They documented composition, per-source weighting, and preprocessing, and showed models trained on this mixture improved on domain-specific evaluation relative to web-only data. It became a widely used training and benchmarking corpus for open models such as GPT-Neo and GPT-J.
UnknownDifficulty 4/10Verified
0882018Distributed Training
Peer reviewed
Paulius Micikevicius, Sharan Narang, Jonah Alben, et al.
The authors stored and computed most values in FP16 while keeping a master copy of the weights in FP32 for stable updates, and introduced loss scaling to shift small gradient values into FP16's representable range. Combined with FP32 accumulation for reductions, this matched full-precision accuracy across image, speech, language, and GAN models. The technique cut memory footprint and doubled arithmetic throughput on hardware with 16-bit units, becoming a default ingredient in large-scale training.
UnknownDifficulty 4/10Verified
0892017Alignment & Preference Learning
Peer reviewed
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, Dario Amodei
The work has people repeatedly pick which of two short video clips of an agent looks closer to a goal, and fits a reward predictor to those choices while a policy is trained against that predictor. Because labeling comparisons is cheaper than demonstrating or coding rewards, it learned tasks like Atari games and simulated-robot backflips from under an hour of human feedback on a small fraction of the agent's interactions. This comparison-based reward modeling became the template later scaled to language-model alignment.
UnknownDifficulty 6/10Verified
0902020Alignment & Preference Learning
Peer reviewed
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel Ziegler, et al.
Instead of training the model to imitate reference summaries, the authors collected human judgments of which of two summaries was better, trained a reward model to predict those judgments, and then used reinforcement learning to make the summarizer score highly on that reward. The resulting summaries were preferred by people over both the reference summaries and models trained by supervised imitation. This work is the practical template for reinforcement learning from human feedback (RLHF) that later underpinned instruction-following chat models.
UnknownDifficulty 6/10Verified
0912023Reasoning & Test-Time Compute
Peer reviewed
Xuezhi Wang, Jason Wei, Dale Schuurmans, et al.
Self-consistency samples multiple diverse reasoning chains for the same question and then marginalizes over the reasoning to pick the answer most paths agree on. Because a correct answer tends to be reachable by several distinct valid derivations while errors are scattered, voting over sampled chains raised accuracy on arithmetic and commonsense benchmarks over standard chain-of-thought. It became a simple, decoding-time add-on for more reliable reasoning at the cost of extra samples.
UnknownDifficulty 5/10Verified
0922023Reasoning & Test-Time Compute
Peer reviewed
Karl Cobbe, John Schulman, Hunter Lightman, Ilya Sutskever, Jan Leike
The authors had humans label the correctness of every step in a model's chain of reasoning, then trained a verifier to score solutions step by step. Ranking many candidate solutions by this process-supervised verifier picked correct answers more often than a verifier trained only on whether the final answer was right. They also released PRM800K, a large dataset of step-level human labels, making process-based reward modeling reproducible for reasoning tasks.
UnknownDifficulty 6/10Verified
0932024Reasoning & Test-Time Compute
System card
OpenAI
Rather than answering immediately, these models generate an extended internal reasoning process and can allocate more or less thinking time to a query. Accuracy on hard reasoning benchmarks improved as more inference compute was spent, establishing test-time compute as a scaling axis distinct from making the model or its training set larger. The reasoning details were released through system cards rather than a formal paper, and the internal chain of thought was kept hidden from users.
UnknownDifficulty 7/10Verified
0942023Agents & Tool Use
Peer reviewed
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, et al.
The model samples candidate API calls (calculator, search, calendar, question-answering, translation) inside ordinary text, executes them, and keeps only the calls whose returned results lower the perplexity of the following tokens, producing a training set it then fine-tunes on. As a result a 6.7B-parameter model learns to invoke tools where they help and gains on arithmetic, factual QA, and temporal tasks without degrading its core language modeling. The mechanism showed tool use can be bootstrapped from the model's own predictions rather than curated examples.
UnknownDifficulty 5/10Verified
0952024Agents & Tool Use
Peer reviewed
John Yang, Carlos E. Jimenez, Ofir Press, Karthik Narasimhan, Xingyao Wang, Graham Neubig
These systems give the model a defined set of actions for navigating files, editing code, and running commands, plus the feedback it gets back from executing those actions, so it can work through a real bug or feature request over many steps. On the SWE-bench benchmark of actual GitHub issues, this agent-plus-tools approach resolved a meaningful fraction of tasks that require reading a codebase, making changes, and checking them. The work showed that the design of the agent's action interface, not just the underlying model, strongly affects how many software tasks get solved, and the released open frameworks made this style of coding agent broadly available.
UnknownDifficulty 6/10Verified
0962022Multimodality
Peer reviewed
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al.
Frozen vision and language backbones are bridged by a Perceiver-style resampler that turns variable numbers of image features into a fixed set of tokens, plus gated cross-attention layers inserted into the language model so text generation can attend to images. Training on large interleaved image-text web data teaches the bridge without retraining the expensive backbones. This allowed few-shot performance on captioning and visual question answering by showing examples in the prompt, rather than fine-tuning a separate model per task.
UnknownDifficulty 6/10Verified
0972023Multimodality
Peer reviewed
Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee
Image features from a frozen CLIP encoder are mapped into the language model's token space by a single trainable projection, so the model can treat an image as extra context alongside a text prompt. The team used a text-only GPT model to generate multimodal instruction-following examples (questions, descriptions, and reasoning about images) and fine-tuned on them. This showed that a capable image-chat assistant could be built cheaply from existing open components plus synthetic instruction data, and it became a common baseline architecture for open multimodal models.
UnknownDifficulty 4/10Verified
0982022Multimodality
Peer reviewed
Robin Rombach, Patrick Esser, Björn Ommer, William Peebles, Saining Xie
An autoencoder first compresses images into a smaller latent representation, and the denoising diffusion model learns to generate in that latent space rather than over millions of raw pixels. A cross-attention mechanism injects conditioning such as text prompts, and the decoder turns the final latent back into a full-resolution image. Working in the smaller space made training and inference cheap enough to run on consumer GPUs, which enabled the open release of a general-purpose text-to-image model.
UnknownDifficulty 6/10Verified
0992024Multimodality
System card
Google DeepMind, OpenAI
The report describes models built to accept and reason over mixed inputs (text, images, audio, and video) natively and to produce text and image outputs, trained jointly across modalities and offered in sizes from on-device to datacenter scale. Native multimodal training is meant to avoid the information loss of connecting independently trained encoders after the fact. The report reports benchmark results across language, coding, reasoning, and multimodal tasks; the underlying weights and full training details are not public, and related closed systems like GPT-4V and GPT-4o pursue similar goals.
UnknownDifficulty 7/10Verified
1002020Retrieval & Memory
Peer reviewed
Vladimir Karpukhin, Barlas Oguz, Sewon Min, et al.
Two BERT encoders map questions and passages into a shared vector space, trained so a question sits near passages that answer it, using in-batch negatives to make training efficient. At query time the question is embedded once and nearest passages are found by fast vector similarity search over a precomputed index. Replacing term-overlap retrieval with learned semantic matching improved retrieval and downstream answer accuracy, and established the dense-retriever design widely used in retrieval-augmented pipelines.
UnknownDifficulty 5/10Verified
1012019Retrieval & Memory
Peer reviewed
Nils Reimers, Iryna Gurevych, Tianyu Gao, Danqi Chen, Liang Wang, Furu Wei, Shitao Xiao, Zheng Liu
Standard BERT compares two sentences by feeding them together, so finding the best match in a large collection needs a separate forward pass per pair, which is far too slow. SBERT instead passes each sentence through a shared BERT with pooling to get one fixed vector per sentence, trained on natural-language-inference and similarity data so that similar meanings give nearby vectors. This lets you embed a corpus once and then compare sentences with cheap vector operations, enabling clustering, semantic search, and duplicate detection at scale.
UnknownDifficulty 4/10Verified
1022021Code Models
Technical report
Mark Chen, Jerry Tworek, Heewoo Jun, et al.
The model is a GPT language model further trained on a large corpus of GitHub code so it can turn a function signature and docstring into a working implementation. Rather than scoring generated code by similarity to a reference, HumanEval defines 164 hand-written problems with unit tests and reports pass@k, the chance that at least one of k sampled solutions passes all tests. This functional metric, plus the finding that sampling many candidates raises success rates, shaped how code-generation models are evaluated and underpinned tools like GitHub Copilot.
UnknownDifficulty 5/10Verified
1032023Open-Weight Model Families
Technical report
Albert Q. Jiang, Alexandre Sablayrolles, Guillaume Lample, Arthur Mensch, Timothée Lacroix, William El Sayed
Mistral 7B combines grouped-query attention (which shares key/value projections across query heads to cut memory and speed up inference) with sliding-window attention (which limits each token's attention to a fixed recent window so longer sequences stay cheap). With these changes it performed on par with or better than models roughly twice its size on common benchmarks. Released with downloadable weights under a permissive Apache 2.0 license, it became a practical base for local deployment and fine-tuning.
Open weightDifficulty 5/10Verified
1042024Open-Weight Model Families
Technical report
Jinze Bai, An Yang, Junyang Lin, Jingren Zhou
Qwen2.5 scales up the pretraining data to roughly 18 trillion tokens and refines the post-training pipeline (supervised fine-tuning plus reinforcement learning from preferences) across a range of model sizes from small to large. It targets improvements in instruction following, structured output, mathematics, and coding, and supports long context windows and many languages. The base and instruction-tuned weights are downloadable, giving practitioners a broad menu of sizes to fit different hardware budgets; most sizes are released under a permissive license while some larger ones use a more restrictive one, so it is open-weight.
Open weightDifficulty 5/10Verified
1052022Multimodality
Dataset paper
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ludwig Schmidt
LAION-5B is a filtered, CLIP-scored collection of billions of image-caption pairs scraped from the web and released openly by the LAION non-profit. It supplied the training data behind Stable Diffusion and many open CLIP and vision-language models, shifting a major bottleneck -- data -- from proprietary to public. It also foregrounded the governance, consent, and safety questions that come with web-scale datasets.
UnknownDifficulty 4/10Verified
1062022Inference & Serving
Peer reviewed
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, et al.
Traditional batching runs a whole batch of requests together until all finish, so short requests wait for long ones and the GPU sits underused. Orca schedules at the granularity of a single decoding iteration: after each token step it can inject newly arrived requests and remove completed ones, keeping the batch continuously full. Combined with selective batching that handles operations with different sequence lengths correctly, this delivered large throughput and latency gains and became the standard serving pattern later adopted by systems like vLLM.
UnknownDifficulty 5/10Verified
1072023Inference & Serving
Peer reviewed
Yaniv Leviathan, Yossi Matias, Charlie Chen, Tianle Cai, Tri Dao, Yuhui Li, Hongyang Zhang
Autoregressive generation is slow because each token requires a full forward pass of the large model, run one at a time. Speculative decoding runs a cheap draft model to guess a short run of upcoming tokens, then the large model checks all of them in a single parallel pass and accepts the longest correct prefix, resampling at the first mismatch. A modified acceptance rule guarantees the result matches what the large model would have produced on its own, so it speeds up inference (often 2-3x) with no loss in output quality.
UnknownDifficulty 6/10Verified
1082022Inference & Serving
Peer reviewed
Elias Frantar, Dan Alistarh, Ji Lin, Song Han, Guangxuan Xiao, Tim Dettmers, Luke Zettlemoyer
GPTQ quantizes a pretrained model's weights layer by layer after training, using an approximation of the layer's Hessian to decide how to round each weight so that the overall error is minimized, and updating the remaining weights to compensate as it goes. This brings models down to roughly 3-4 bits per weight in one pass without retraining, shrinking memory enough to fit very large models on a single GPU. Related methods such as SmoothQuant (which shifts activation outliers into weights) and AWQ (which protects the weights most important to activations) address the same goal of low-bit inference through different trade-offs.
UnknownDifficulty 6/10Verified
1092022Long Context & Efficient Sequences
Peer reviewed
Albert Gu, Karan Goel, Christopher Ré
State space models describe a sequence through a continuous linear recurrence, but naive versions are numerically unstable and too slow to train. S4 reparameterizes the state matrix in a structured diagonal-plus-low-rank form that can be computed as a convolution, making training stable and efficient while scaling near-linearly with sequence length. This let a single architecture model dependencies over tens of thousands of steps, strongly outperforming transformers on long-range benchmarks and seeding the later line of work leading to Mamba.
UnknownDifficulty 7/10Verified
1102024Long Context & Efficient Sequences
Peer reviewed
Albert Gu, Tri Dao
The authors made structured state-space model parameters input-dependent so the model can selectively remember or forget information along a sequence, then designed a hardware-aware parallel scan that computes the recurrence efficiently on GPUs without materializing the full state. Mamba runs in linear time with constant memory per step at inference and handles very long sequences, while matching or exceeding Transformers of similar size on language, audio, and genomics. It offered an attention-free architecture with higher inference throughput for long-context modeling.
UnknownDifficulty 7/10Verified
1112021Evaluation & Benchmarks
Benchmark paper
Dan Hendrycks, Collin Burns, Steven Basart, et al.
The benchmark collects roughly 16,000 exam-style questions at difficulties from high school to professional level and scores models without task-specific training. At release most models scored near the 25% random-chance baseline while the largest GPT-3 reached about 44%, with especially weak and poorly calibrated performance on subjects like law and morality. It became a standard headline number for comparing the general knowledge of frontier models.
UnknownDifficulty 4/10Verified
1122021Evaluation & Benchmarks
Benchmark paper
Karl Cobbe, Vineet Kosaraju, John Schulman, Dan Hendrycks, Jacob Steinhardt
MATH contains 12,500 competition-level problems, each with a worked solution, spanning subjects like algebra, geometry, and number theory at varying difficulty. Because answers require multi-step derivations rather than lookup, early large models scored very low, making it a clear measure of reasoning progress. It is often paired with GSM8K, a grade-school word-problem set, and together they became standard yardsticks for tracking improvements in chain-of-thought and mathematical reasoning.
UnknownDifficulty 4/10Verified
1132021Evaluation & Benchmarks
Benchmark paper
Mark Chen, Wojciech Zaremba, Jacob Austin, Augustus Odena, Charles Sutton
HumanEval (164 problems, from the Codex paper, arXiv 2107.03374) and MBPP (about 970 crowd-sourced entry-level problems, arXiv 2108.07732) each pair a natural-language prompt with test cases, and evaluate a model by whether sampled completions pass all tests. They introduced the pass@k metric, which estimates the chance that at least one of k sampled programs is correct, giving a functional rather than lexical measure of code generation. The two became the default yardsticks for code LLMs and the basis for later, harder coding benchmarks.
UnknownDifficulty 4/10Verified
1142023Evaluation & Benchmarks
Benchmark paper
Lianmin Zheng, Wei-Lin Chiang, Ion Stoica, Xuechen Li, Yann Dubois, Tatsunori B. Hashimoto
Chatbot Arena collects anonymous head-to-head comparisons: users chat with two unnamed models, vote for the better response, and the votes feed an Elo-style rating that ranks models. MT-Bench is a set of multi-turn questions scored by a strong model acting as an automated judge, which the paper validates against human preferences while also documenting judge biases such as favoring longer answers or the first response shown. Together they gave the field a way to evaluate conversational quality and instruction following that static multiple-choice benchmarks cannot capture.
UnknownDifficulty 4/10Verified
1152023Evaluation & Benchmarks
Peer reviewed
Oscar Sainz, Zhang, et al.
The work documents three linked failures in how models are scored: contamination, where test items leak into training data so results overstate ability; saturation, where top models cluster near the ceiling of older benchmarks so the numbers no longer separate them; and judge bias, where using a model as an automated grader introduces systematic errors like preferring verbose or self-similar answers. By characterizing these effects, it argues that many headline evaluation numbers are unreliable and motivates contamination checks, harder benchmarks, and more careful judging protocols.
UnknownDifficulty 5/10Verified
1162020Interpretability & Safety
Technical report
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter
The essay lays out three claims: networks contain features (directions that detect meaningful concepts), features are connected by weighted circuits that implement algorithms, and analogous features and circuits recur across models. Working through vision networks, it shows curve detectors and other units that can be read and tested. It set the agenda and vocabulary for the mechanistic-interpretability program that followed.
UnknownDifficulty 6/10Verified
1172021Interpretability & Safety
Technical report
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Chris Olah
The report decomposes a Transformer into a residual stream that attention heads read from and write to, and shows how heads compose across layers to move and combine information. It introduces tools like the QK and OV circuits and identifies simple mechanisms in small models. This gave mechanistic interpretability a concrete framework for studying language models.
UnknownDifficulty 7/10Verified
1182022Interpretability & Safety
Technical report
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Chris Olah
By tracking model behavior over training, the authors found that a phase change in in-context learning coincides with the formation of induction heads that look back for a previous occurrence of the current token and copy what followed. They present converging evidence that these heads drive much of a model’s ability to learn from its prompt. It connected a specific circuit to an emergent capability.
UnknownDifficulty 7/10Verified
1192022Interpretability & Safety
Technical report
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Chris Olah
Using small toy models, the authors demonstrate that when features are sparse a network represents many of them as overlapping directions rather than one-per-neuron, trading interference for capacity. They map when superposition appears and how it organizes into geometric structures. This reframed why interpretability is hard and motivated methods to pull features apart.
UnknownDifficulty 7/10Verified
1202023Interpretability & Safety
Technical report
Trenton Bricken, Adly Templeton, Joshua Batson, Chris Olah
The work trains a sparse autoencoder on the activations of a small language model and recovers thousands of features that each correspond to a single, nameable concept, resolving the superposition problem in practice. The features are interpretable, can be steered, and generalize. It became the template for the sparse-autoencoder interpretability work now applied to frontier models.
UnknownDifficulty 7/10Verified
1212020Scaling Laws & Compute
Essay
Gwern Branwen
Written after GPT-3, the essay collects the evidence that loss and many capabilities improve predictably with size, and contends that the field had underrated how far pure scaling would go. It frames scaling as a hypothesis to be taken seriously rather than a curiosity. It is one of the defining statements of the scaling-era worldview and its stance is debated.
UnknownDifficulty 4/10Verified
1222009Data, Corpora & Tokenization
Peer reviewed
Alon Halevy, Peter Norvig, Fernando Pereira
Drawing on web-scale examples, the authors contend that unreasonably large corpora let comparatively simple methods succeed where elaborate ones with little data fail, and urge researchers to embrace data. Written years before deep learning’s dominance, it foreshadowed the data-centric logic later formalized by scaling laws. It is a conceptual ancestor of the scaling era.
UnknownDifficulty 4/10Verified
1232021Evaluation & Benchmarks
Critique
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, Margaret Mitchell
The paper contends that scaling training data and compute concentrates benefits, raises environmental and labor costs, encodes and amplifies bias from unaudited web text, and produces systems that stitch together form without meaning (the “stochastic parrot”). It calls for documentation, curation, and restraint. It became the defining critical counterpoint to the scale-first paradigm and remains contested.
UnknownDifficulty 4/10Verified
1242021Multimodality
Preprint
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray
DALL-E encodes images as discrete tokens and trains a large Transformer to model the sequence of text followed by image tokens, then samples images from captions and reranks them with CLIP. It produced novel, compositional images without task-specific training. It opened the modern text-to-image line later dominated by diffusion methods.
UnknownDifficulty 6/10Verified
1252022Multimodality
Preprint
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman
Whisper is an encoder-decoder Transformer trained on hundreds of thousands of hours of diverse audio-text pairs scraped from the web, covering many languages and tasks in one model. It approaches specialized systems zero-shot and is resilient to accents and noise. Its open release made strong speech recognition broadly available.
UnknownDifficulty 5/10Verified
1262017Continual Learning & Memory
Peer reviewed
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Raia Hadsell, Demis Hassabis
The method estimates how sensitive prior-task performance is to each weight using the Fisher information, then adds a quadratic penalty anchoring the important weights while leaving the rest free to adapt. A single network could learn a sequence of tasks and retain earlier ones where ordinary training would overwrite them. It turned catastrophic forgetting from a vague failure into a measurable, tractable problem and anchored the continual-learning literature.
UnknownDifficulty 6/10Verified
1272014Neural Foundations
Peer reviewed
Jeffrey Pennington, Richard Socher, Christopher Manning
GloVe builds a matrix counting how often each word appears near each other word across the whole corpus, then trains word vectors so that the dot product of two vectors approximates the logarithm of their co-occurrence count, weighting frequent pairs so common words do not dominate. This uses aggregate statistics directly rather than scanning context windows one at a time like word2vec. The resulting vectors placed related words near each other and encoded analogies as vector offsets, and became a widely used pretrained input for later NLP models.
UnknownDifficulty 4/10Verified
1282015Neural Foundations
Peer reviewed
Minh-Thang Luong, Hieu Pham, Christopher Manning
The paper lets a translation decoder, at each output step, compute a weighted average of encoder hidden states where the weights come from a similarity score between the current decoder state and each source state, then combines that context with the decoder state to predict the next word. It compared scoring functions (dot product, general, concat) and introduced a 'local' attention that only looks at a small window of source positions to save computation. These variants improved translation quality and clarified the design space, and the dot-product scoring it favored became standard in later attention and Transformer work.
UnknownDifficulty 5/10Verified
1291989Neural Foundations
Peer reviewed
George Cybenko, Kurt Hornik
Cybenko (1989) showed this for sigmoid-type activations and Hornik (1991) generalized it to broad classes of activation functions, proving that finite sums of scaled, shifted activations are dense in the space of continuous functions. The result is an existence guarantee: for any target accuracy some network with enough hidden units achieves it. It does not say how many units are needed or how to find the weights by training, but it removed the theoretical objection that networks might be fundamentally incapable of representing complex functions.
UnknownDifficulty 5/10Verified
1301990Neural Foundations
Peer reviewed
Jeffrey Elman
The network copies its hidden-layer state at each step into a 'context' layer that is fed back in alongside the next input, so the hidden units learn representations that depend on prior elements of the sequence. Trained on tasks like predicting the next word or discovering word boundaries in a stream, it developed internal representations that reflected grammatical categories and structure without being told them. This showed that temporal structure could be handled by recurrence rather than by explicit time-delay windows, and the architecture became a foundation for later recurrent networks.
UnknownDifficulty 4/10Verified
1311994Neural Foundations
Peer reviewed
Sepp Hochreiter, Yoshua Bengio, Patrice Simard, Paolo Frasconi
The paper showed mathematically that for a recurrent network to store information robustly over time its dynamics must contract, but contraction causes error gradients passed backward through many time steps to shrink toward zero, so distant past inputs contribute almost nothing to weight updates. When the dynamics instead expand, gradients blow up. This trade-off means standard gradient descent cannot reliably link causes and effects separated by many steps. The analysis motivated later remedies such as gating (LSTM, GRU), gradient clipping, and better initialization.
UnknownDifficulty 5/10Verified
1322014Neural Foundations
Peer reviewed
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, Yoshua Bengio
One recurrent network (the encoder) reads a source sentence and compresses it into a fixed-length vector, and a second recurrent network (the decoder) generates the target sentence from that vector, trained jointly to maximize the probability of the correct output. To make the recurrence trainable over longer spans the paper introduced the GRU, which uses update and reset gates to control how much past state is kept or overwritten, achieving LSTM-like behavior with a simpler structure. The encoder-decoder setup became the template for neural machine translation and sequence-to-sequence learning, and its learned phrase representations improved a statistical translation system.
UnknownDifficulty 4/10Verified
1332011Neural Foundations
Peer reviewed
Ronan Collobert, Jason Weston, Léon Bottou, et al.
The system maps words to learned vector embeddings, processes sentences with a convolutional network, and is trained jointly across tasks like part-of-speech tagging, chunking, named-entity recognition, and semantic role labeling. Crucially it pretrains word embeddings on large unlabeled corpora using a ranking objective, then reuses them, so the model discovers useful features rather than relying on task-specific hand-crafted inputs. It reached competitive accuracy across tasks with one architecture, demonstrating the transfer value of unsupervised pretrained word representations that later work built on.
UnknownDifficulty 5/10Verified
1341993Neural Foundations
Peer reviewed
Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, Robert L. Mercer
The paper frames translation as finding the target sentence most probable given the source, decomposed via Bayes' rule into a translation model and a language model, and introduces a hidden alignment variable saying which source word generated each target word. The five IBM models add successive detail (word translation probabilities, then absolute position, fertility of how many words a source word produces, and distortion of position), and their parameters are estimated from unaligned sentence pairs using the EM algorithm. These alignment models became the statistical foundation of machine translation for roughly two decades and the basis for later phrase-based systems.
UnknownDifficulty 6/10Verified
1351995Neural Foundations
Peer reviewed
Gerald Tesauro
The program uses a neural network to estimate the expected outcome of a board position and trains it with TD(lambda), which adjusts predictions so that each position's estimated value moves toward the value of positions that follow it during play. By playing hundreds of thousands of games against itself it improved without expert-labeled positions or a hand-tuned evaluation function. It reached strong tournament-level play and even influenced human opening theory, providing an early demonstration that reinforcement learning combined with function approximation could master a large-state-space game.
UnknownDifficulty 5/10Verified
1362014Neural Foundations
Preprint
Alex Graves, Greg Wayne, Ivo Danihelka, Jason Weston, Sumit Chopra, Antoine Bordes
A controller network emits read and write operations addressed to memory locations using soft, differentiable attention weights (based on content similarity and location shifting), so gradients can flow through the memory access and the model learns how to use its memory end-to-end. This separates computation from an addressable storage bank, letting the network learn procedures like copying, sorting, and associative recall and generalize them to longer inputs than seen in training. It, alongside Memory Networks, showed that networks augmented with external memory could learn algorithm-like behavior, influencing later memory-augmented and attention-based models.
UnknownDifficulty 6/10Verified
1372012Neural Foundations
Peer reviewed
Geoffrey Hinton, Li Deng, Dong Yu, et al.
In a speech recognizer the acoustic model estimates how likely each short audio frame corresponds to each sub-phonetic state; the paper describes replacing the standard Gaussian-mixture estimator with a deep neural network that takes several frames of audio features and outputs those state probabilities, often pretrained layer by layer and then fine-tuned. Reported jointly by teams at Microsoft, Google, IBM, and the University of Toronto, the deep networks lowered word error rates on multiple large-vocabulary benchmarks. The shared results gave the field consistent evidence that deep acoustic models were a general improvement, and they were adopted in commercial speech systems.
UnknownDifficulty 5/10Verified
1381969Origins & Computability
Critique
Marvin Minsky, Seymour Papert
Through geometric and algebraic analysis the authors characterized exactly which predicates a single-layer perceptron can and cannot represent, showing that some require information the local, limited-order perceptron cannot combine, with XOR and connectedness as prominent examples of failures. They noted that multilayer networks could in principle overcome these limits but that no effective training method for them was then known. The rigorous negative results are widely credited with cooling early enthusiasm and funding for neural network research until multilayer training via backpropagation was popularized in the 1980s.
UnknownDifficulty 5/10Verified
1392021Transformer Architecture
Peer reviewed
Ofir Press, Noah A. Smith, Mike Lewis
Instead of adding position vectors to token embeddings, ALiBi biases each attention score downward in proportion to how far apart the two tokens are, with a per-head slope. Because this bias is a simple distance function rather than a lookup limited to training lengths, a model trained on sequences of, say, 1024 tokens keeps working when evaluated on far longer inputs. This made length extrapolation cheap and removed the need to train at the target context length to get usable long-context behavior.
UnknownDifficulty 5/10Verified
1402018Transformer Architecture
Peer reviewed
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani, Zihang Dai, Zhilin Yang, Quoc V. Le, Ruslan Salakhutdinov
Standard Transformers split long text into fixed segments processed independently, so the model never sees dependencies crossing a segment boundary. Transformer-XL caches the hidden states of the previous segment and lets the current segment attend back into them, while switching from absolute to relative position encodings so the reused states stay positionally consistent. This extended the effective context well beyond a single segment and sped up evaluation, enabling coherent modeling of longer-range dependencies.
UnknownDifficulty 6/10Verified
1412020Encoders
Peer reviewed
Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning
Rather than masking tokens and predicting them, ELECTRA uses a small generator to swap some tokens for plausible alternatives, then trains the main model to decide, for every token, whether it was replaced. Because the loss covers all positions rather than the small masked subset, each training step yields more signal per example. This let ELECTRA match or exceed BERT-scale accuracy using substantially less pretraining compute, making strong encoders reachable on smaller budgets.
UnknownDifficulty 5/10Verified
1422020Encoder–Decoders
Peer reviewed
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer
BART corrupts documents with noise such as token masking, deletion, sentence shuffling, and text-span infilling, then trains a bidirectional encoder plus autoregressive decoder to reconstruct the original. This seq2seq setup means the same pretrained model handles classification (via the encoder) and generation like summarization and translation (via the decoder), rather than needing separate architectures. Text infilling in particular proved effective, and BART reached strong results on generation benchmarks while remaining competitive on discriminative tasks.
UnknownDifficulty 5/10Verified
1432021Encoders
Peer reviewed
Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen
DeBERTa keeps a token's content and its relative position as two distinct vectors and computes attention weights from content-to-content, content-to-position, and position-to-content terms, capturing that a word's relationship to another depends on their distance. Because relative encoding alone loses absolute placement needed for tasks like masked-word prediction, it adds an enhanced mask decoder that folds absolute positions back in just before the output layer. These changes improved sample efficiency and accuracy on language-understanding benchmarks relative to BERT and RoBERTa at comparable model sizes.
UnknownDifficulty 6/10Verified
1442022Scaling Laws & Compute
Peer reviewed
Jason Wei, Yi Tay, Rishi Bommasani, et al.
The authors survey tasks where accuracy stays near chance for small models and then rises sharply beyond a certain parameter or compute scale, calling these emergent abilities. They catalog examples across few-shot prompting and augmented-prompting settings, showing the jumps are not evident from the performance trend of smaller models. The claim shaped how the field reasoned about scaling, suggesting some capabilities cannot be forecast and appear only after crossing a size threshold.
UnknownDifficulty 5/10Verified
1452023Scaling Laws & Compute
Peer reviewed
Rylan Schaeffer, Brando Miranda, Sanmi Koyejo
Schaeffer et al. show that metrics like exact-match or multiple-choice accuracy are harsh and nonlinear: a model's underlying per-token performance can improve smoothly while the reported score stays flat then spikes once the whole answer becomes correct. Re-scoring the same models and tasks with continuous or partial-credit metrics converts many reported emergent curves into steady, extrapolable trends. The work reframes much claimed emergence as a measurement choice, cautioning that metric selection, not new capability, can manufacture the appearance of a threshold.
UnknownDifficulty 5/10Verified
1462023Scaling Laws & Compute
Peer reviewed
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, et al.
The authors trained hundreds of models while holding the amount of unique text fixed and varying how many times it was repeated, then fit scaling laws that treat repeated tokens as worth less than fresh ones. They found that repeating data for up to about four epochs yields loss nearly identical to using that much new data, with returns decaying quickly after roughly 16 epochs. This gave data-limited teams a principled way to allocate compute between more epochs and more parameters, and quantified when scraping or generating more data stops helping.
UnknownDifficulty 6/10Verified
1472022Mixture-of-Experts
Peer reviewed
Nan Du, Yanping Huang, Andrew M. Dai, et al.
GLaM replaces the feed-forward block in every other transformer layer with a set of 64 experts and a learned gate that routes each token to the top two, so total capacity grows without activating all weights per token. This decoupled model size from per-token compute, allowing a much larger parameter count at fixed inference cost. The result was competitive or better zero/one/few-shot accuracy than a dense 175B model at substantially lower compute and energy, demonstrating sparse MoE as a practical scaling path for large LMs.
UnknownDifficulty 6/10Verified
1482022Mixture-of-Experts
Peer reviewed
Yanqi Zhou, Tao Lei, Hanxiao Liu, et al.
In standard MoE each token picks its top experts, which causes uneven loads where some experts overflow and drop tokens. Expert Choice inverts this: every expert picks a fixed quota of the tokens it scores highest, so all experts stay exactly full and a token may be handled by a variable number of experts. This removed the need for load-balancing loss terms and capacity-factor tuning, and trained faster to a given quality (reported over 2x convergence speedup) while improving downstream results at matched compute.
UnknownDifficulty 6/10Verified
1492022Mixture-of-Experts
Preprint
Barret Zoph, Irwan Bello, Sameer Kumar, et al.
The authors traced MoE training instabilities to large router logits and showed that a z-loss penalizing those logit magnitudes stabilizes training without hurting quality. They also identified fine-tuning pitfalls, such as sparse and dense layers preferring different hyperparameters and expert dropout settings, and gave concrete recipes to address them. Combining these, their ST-MoE-32B model trained stably and transferred well across many NLP tasks, turning sparse experts from a finicky research artifact into a reproducible design.
UnknownDifficulty 6/10Verified
1502024Mixture-of-Experts
Technical report
Damai Dai, Chengqi Deng, Chenggang Zhao, et al.
DeepSeekMoE splits each expert into smaller pieces and increases their number so the router can compose more precise combinations, and it isolates a handful of shared experts that every token uses to capture redundant, general knowledge. This reduces the parameter redundancy and poor specialization seen in conventional MoE with a few large experts. At matched activated compute it matched or beat larger dense and standard-MoE baselines, and the design underpinned the efficiency of later DeepSeek models.
UnknownDifficulty 6/10Verified
1512023Data, Corpora & Tokenization
Dataset paper
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al.
The authors built a pipeline (MacroData Refinement) that applies URL filtering, language identification, trafilatura-based text extraction, quality heuristics, and both fuzzy and exact deduplication to Common Crawl at scale. From this they released RefinedWeb, a five-trillion-token web-only corpus, and trained Falcon models that matched or exceeded models trained on curated mixtures. This demonstrated that scale plus rigorous cleaning of raw web data can substitute for hand-picked high-quality sources, and provided a large open dataset for the community.
UnknownDifficulty 5/10Verified
1522023Data, Corpora & Tokenization
Technical report
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, et al.
The authors assembled a filtered set of high-educational-value code from the web plus GPT-generated textbook-style text and problem/solution exercises, then trained phi-1, a 1.3B model, on only about 7 billion tokens. Despite being orders of magnitude smaller in data and parameters than contemporaries, phi-1 reached strong pass@1 on HumanEval and MBPP. This provided evidence that carefully curated, instruction-dense data can dramatically improve sample efficiency, seeding the small-high-quality-data line of work.
UnknownDifficulty 5/10Verified
1532024Data, Corpora & Tokenization
Peer reviewed
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson
The authors trained models on data produced by earlier model generations and observed that each generation's outputs narrow toward the mean, losing rare events and low-probability tails until later models converge to a degenerate distribution. They gave a theoretical account attributing this to compounding statistical approximation, functional expressivity, and sampling errors across generations. The finding warned that as AI-generated text saturates the web, indiscriminate training on it degrades future models, making access to genuine human-produced data increasingly valuable.
UnknownDifficulty 5/10Verified
1542019Distributed Training
Peer reviewed
Yanping Huang, Youlong Cheng, Ankur Bapna, et al.
The model layers are divided into consecutive stages placed on separate accelerators, and each minibatch is broken into micro-batches that flow through the stages so multiple devices compute at once instead of waiting idle. Gradients are accumulated across micro-batches and applied synchronously, so the result is numerically identical to non-pipelined training. Combined with activation recomputation, this let models with billions of parameters be trained across devices that individually could not hold them, at near-linear throughput scaling.
UnknownDifficulty 5/10Verified
1552018Distributed Training
Infrastructure paper
Noam Shazeer, Youlong Cheng, Niki Parmar, Yuanzhong Xu, HyoukJoong Lee, Blake Hechtman, Zhifeng Chen
Building on Mesh-TensorFlow's idea of mapping tensor dimensions onto a logical device mesh, GSPMD takes per-tensor sharding annotations, propagates them through the whole XLA computation graph, and inserts the needed collective communication. The same code can therefore be run data-parallel, tensor/operator-parallel, or pipeline-parallel just by changing annotations. This decoupled how a model is written from how it is partitioned, enabling parallelization of very large models with minimal code changes across thousands of accelerators.
UnknownDifficulty 6/10Verified
1562023Distributed Training
Infrastructure paper
Yanli Zhao, Andrew Gu, Rohan Varma, et al.
Instead of replicating the full model on every worker as standard data parallelism does, FSDP splits parameters, gradients, and optimizer state into shards held by different workers; before a layer runs it all-gathers that layer's parameters, then frees them afterward, and reduce-scatters gradients during the backward pass. Communication is overlapped with computation and grouped into units to limit memory spikes and latency. Delivered as a native PyTorch API, it let practitioners train models with hundreds of billions of parameters on commodity clusters without a separate framework.
UnknownDifficulty 5/10Verified
1572016Distributed Training
Preprint
Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin
During the forward pass only a sparse set of checkpoint activations is kept and the rest are dropped; during backpropagation the missing activations are recomputed on demand from the nearest checkpoint. Placing checkpoints roughly every sqrt(n) layers reduces activation memory to O(sqrt(n)) at the cost of one extra forward computation. This gradient/activation checkpointing let much deeper networks and longer sequences be trained within a fixed GPU memory budget, and became a standard building block in later large-model training systems.
UnknownDifficulty 4/10Verified
1582023Distributed Training
Technical report
Houwen Peng, Kan Wu, Han Hu, Peng Cheng, Paulius Micikevicius
The framework extends mixed-precision training so that not just matrix multiplies but also gradients, the optimizer's moment states, and distributed communication use FP8, guarded by automatic per-tensor scaling to keep values inside FP8's narrow dynamic range and by decoupling precision across components to avoid underflow. Applied to GPT-scale models, it matched the accuracy of BF16 training while cutting memory footprint and communication volume and raising throughput. This pushed practical LLM training below 16 bits across more of the pipeline than earlier FP8 schemes, which had limited FP8 to the matmuls.
UnknownDifficulty 6/10Verified
1592024Alignment & Preference Learning
Peer reviewed
Mohammad Gheshlaghi Azar, Rémi Munos, Kawin Ethayarajh, Douwe Kiela, Jiwoo Hong, James Thorne, Yu Meng, Danqi Chen
Each variant targets a specific limitation of DPO: IPO replaces the log-sigmoid objective with a bounded one to curb overfitting when preferences are near-deterministic; KTO learns from single labeled examples marked desirable or undesirable rather than requiring paired comparisons; ORPO folds a preference odds-ratio penalty directly into supervised fine-tuning so no separate alignment stage or reference model is needed; SimPO removes the reference model and uses a length-normalized reward with a target margin. Together they made preference-based alignment cheaper and more stable, reducing reliance on paired data, reference models, and reward-model training.
UnknownDifficulty 6/10Verified
1602022Alignment & Preference Learning
Peer reviewed
Leo Gao, John Schulman, Jacob Hilton, Mrinank Sharma, Ethan Perez
The authors trained policies against reward models of varying size and data, then compared the proxy reward the model assigned against a gold reward model treated as ground truth. As optimization pressure (measured in KL distance from the initial policy) increased, proxy reward kept rising while true reward peaked and then fell, and they fit scaling laws for where this divergence begins. This gave RLHF practitioners a concrete way to predict when a reward model stops being a trustworthy target and to bound optimization accordingly, explaining downstream failures like sycophancy where the policy exploits reward-model quirks rather than improving.
UnknownDifficulty 6/10Verified
1612018Alignment & Preference Learning
Preprint
Geoffrey Irving, Paul Christiano, Dario Amodei, Collin Burns, Jan Leike, Ilya Sutskever
The weak-to-strong experiments used a small model's imperfect labels to fine-tune a much larger pretrained model and found the larger model generalized beyond its teacher's errors, an analogy for humans supervising superhuman systems. Related debate proposals have two models argue opposing positions so a limited judge can adjudicate claims it could not verify directly. Together these define the scalable-oversight problem and give early empirical evidence that supervision signal can transfer even when the supervisor is less capable than the system being trained.
UnknownDifficulty 7/10Verified
1622023Reasoning & Test-Time Compute
Peer reviewed
Denny Zhou, Quoc V. Le, Ed H. Chi, Wenhu Chen, William W. Cohen, Shunyu Yao, Karthik Narasimhan
Tree of Thoughts generalizes chain-of-thought by having the model generate multiple candidate reasoning steps, score them with a value estimate, and search the resulting tree with breadth- or depth-first exploration and backtracking. Least-to-Most decomposes a hard problem into ordered easier subproblems solved in sequence, and Program-of-Thought offloads arithmetic and logic to generated code executed by an interpreter. The shared move is treating reasoning as a structured, evaluable process rather than a single sampled string, which raised accuracy on tasks requiring planning, search, or exact computation.
UnknownDifficulty 6/10Verified
1632023Agents & Tool Use
Product release
Ehud Karpas, Opher Lieber, Barak Lenz, Yoav Levine, Yoav Shoham, OpenAI, Anthropic
MRKL proposed a neuro-symbolic architecture in which a router directs a query to discrete expert modules, such as a calculator, database, or API, and merges their outputs. Later function-calling formats let a model emit a structured call that host code executes and feeds back, and the Model Context Protocol standardizes how tools, data sources, and context are exposed to models across applications. The common contribution is a defined boundary and calling convention between the model and outside systems, which enabled reliable access to current data, exact computation, and actions the model cannot perform internally.
UnknownDifficulty 4/10Verified
1642023Agents & Tool Use
Peer reviewed
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao
After each trajectory, an evaluator produces a reward or error signal, and the agent generates a natural-language reflection on what went wrong, which is appended to an episodic memory buffer that conditions subsequent trials. Because learning happens through text in the context window rather than gradient updates, the same frozen model improves across attempts on decision-making, coding, and reasoning benchmarks. This provided a cheap iterative-improvement mechanism for agents that reuses a model's own language ability as the update rule.
UnknownDifficulty 5/10Verified
1652023Agents & Tool Use
Preprint
Guanzhi Wang, Linxi Fan, Anima Anandkumar, Joon Sung Park, Percy Liang, Michael S. Bernstein
Voyager runs a GPT-4 agent in Minecraft with an automatic curriculum that proposes progressively harder tasks, a growing library of executable code skills it writes and reuses, and an iterative prompting loop that repairs failures, letting it explore without a fixed objective. Generative Agents give each simulated character a memory stream of observations, a retrieval scheme, and periodic reflection that synthesizes higher-level inferences, producing emergent social behavior like spreading information and coordinating events. Together they showed that lifelong skill acquisition and persistent, plausible multi-agent behavior can be driven largely by language models plus structured memory rather than task-specific training.
UnknownDifficulty 5/10Verified
1662021Multimodality
Peer reviewed
Chao Jia, Yinfei Yang, Ye Xia, et al.
ALIGN trains separate image and text encoders to map matched image-caption pairs close together in a shared embedding space using a contrastive loss, learned over ~1.8 billion raw web image/alt-text pairs with only minimal frequency-based cleaning. By tolerating noisy supervision at scale, it removed the bottleneck of building large curated vision-language datasets and enabled strong zero-shot classification and image-text retrieval from web data alone.
UnknownDifficulty 5/10Verified
1672023Multimodality
Peer reviewed
Junnan Li, Dongxu Li, Silvio Savarese, Steven C. H. Hoi, Caiming Xiong
BLIP-2 keeps a pretrained image encoder and a pretrained LLM frozen and trains only a small Q-Former, a set of learnable query tokens that extract the visual features most relevant to text and feed them into the language model. This two-stage bridging cuts the trainable parameters and compute needed for vision-language pretraining by orders of magnitude, letting new image encoders and LLMs be combined into captioning and visual question-answering systems without full end-to-end retraining.
UnknownDifficulty 5/10Verified
1682023Multimodality
Technical report
Jinze Bai, Jingren Zhou, Zhe Chen, Jifeng Dai, Xi Chen, Radu Soricut, Shaohan Huang, Furu Wei
These models connect a strong vision encoder to a language model and train through staged image-text pretraining followed by multimodal instruction tuning, adding capabilities such as higher-resolution input, region grounding, OCR, and multilingual support. Represented by Qwen-VL, the family shipped weights, tooling, and reproducible recipes that made competitive image understanding, document reading, and grounded visual dialogue available to build on directly rather than only behind proprietary services.
UnknownDifficulty 5/10Verified
1692023Multimodality
Peer reviewed
Alexander Kirillov, Ross Girshick, Piotr Dollár, Shilong Liu, Lei Zhang, Rohit Girdhar, Ishan Misra
Segment Anything trains a promptable segmentation model on a billion-mask dataset so that a point, box, or text-linked prompt yields object masks for categories never explicitly labeled; Grounding DINO adds open-vocabulary detection from text queries, and ImageBind aligns six modalities into one embedding space. Represented by SAM, this family removed the need to collect labels and train a fresh model for each new segmentation or detection target, enabling zero-shot and interactive perception that downstream systems can prompt directly.
UnknownDifficulty 5/10Verified
1702020Retrieval & Memory
Peer reviewed
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang
REALM augments masked language-model pretraining with a learned retriever that fetches relevant documents from a corpus and conditions predictions on them, backpropagating the language-modeling signal through the retrieval step so the retriever learns which passages help. This let a relatively small model consult an external, updatable knowledge store instead of memorizing facts in its weights, improving open-domain question answering and making the knowledge source inspectable and swappable.
UnknownDifficulty 6/10Verified
1712021Retrieval & Memory
Peer reviewed
Gautier Izacard, Edouard Grave, Sebastian Borgeaud, Jack W. Rae, Laurent Sifre
These methods retrieve many relevant passages from a large corpus and fuse them into the generator: Fusion-in-Decoder encodes each passage separately and combines them in the decoder, RETRO cross-attends to chunk-level neighbors from a trillion-token database during pretraining, and Atlas jointly trains retriever and reader for few-shot knowledge tasks. Represented by RETRO, the family let models trade parameters for an external datastore, reaching strong language-modeling and question-answering results at lower model size while keeping the knowledge base updatable.
UnknownDifficulty 6/10Verified
1722020Retrieval & Memory
Peer reviewed
Omar Khattab, Matei Zaharia, Urvashi Khandelwal, Mike Lewis, Dan Jurafsky
ColBERT represents each query and document as a bag of contextual token embeddings and computes relevance by summing, for each query token, its maximum similarity to any document token, so most document computation is precomputed offline and only lightweight matching happens at query time. This late-interaction design, alongside related rerankers and nearest-neighbor language models, delivered near cross-encoder ranking quality at a scale and latency close to bag-of-vectors search, making high-quality dense retrieval practical over large collections.
UnknownDifficulty 5/10Verified
1732022Retrieval & Memory
Peer reviewed
Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov
ROME uses causal tracing to identify the mid-layer feed-forward modules that store a given factual association, then applies a rank-one weight edit that rewrites that fact while leaving unrelated knowledge intact; MEMIT extends the same mechanism to insert thousands of edits at once. This gave a targeted, low-cost way to update or correct specific facts in a trained model and provided evidence for where and how factual knowledge is stored inside transformers.
UnknownDifficulty 6/10Verified
1742022Code Models
Peer reviewed
Yujia Li, David Choi, Junyoung Chung, et al.
AlphaCode generates a very large pool of candidate programs per problem (up to millions), then filters them by running the provided example tests and clusters the survivors by their behavior on generated inputs to pick a small number to submit. This sample-filter-cluster loop, rather than a single accurate decode, is what let it reach an average ranking around the top 54% of participants on Codeforces contests. It demonstrated that hard, multi-step reasoning problems could be approached by trading inference-time compute for correctness when an execution oracle is available.
UnknownDifficulty 6/10Verified
1752022Code Models
Peer reviewed
Yue Wang, Steven C.H. Hoi, Erik Nijkamp, Caiming Xiong, Daniel Fried, Mike Lewis, Mohammad Bavarian, Mark Chen
CodeT5 (encoder-decoder, identifier-aware), CodeGen (autoregressive, multi-turn program synthesis), and InCoder (masked-span infilling) explored how objective and architecture shape code models. The FIM paper showed a simple data transformation: move a randomly chosen middle span to the end with sentinel tokens, so a standard causal model learns to fill gaps using surrounding code while retaining normal left-to-right ability. This removed the need for a separate bidirectional model to power editor-style completions like inserting a function body between existing lines.
UnknownDifficulty 5/10Verified
1762023Code Models
Technical report
Raymond Li, Harm de Vries, Leandro von Werra, Baptiste Rozière, Gabriel Synnaeve, Daya Guo, Binyuan Hui
StarCoder was trained on The Stack, a permissively licensed GitHub corpus with PII redaction and an opt-out mechanism, and supports fill-in-the-middle plus an 8K-token window for repository-scale context. Code Llama (continued-pretraining of Llama 2 on code, long-context and instruct variants) and DeepSeek-Coder (repo-level pretraining with strong data curation) extended the same open, code-specialized line. Together they gave practitioners high-quality open-weight coding models with transparent data provenance, enabling local deployment, fine-tuning, and reproducible evaluation.
UnknownDifficulty 5/10Verified
1772022Code Models
Peer reviewed
Bei Chen, Jian-Guang Lou, Weizhu Chen, Xinyun Chen, Denny Zhou
Rather than treating a generated program as final, these methods execute it against unit tests (often model-generated) and feed the resulting pass/fail signals and error traces back to guide selection or iterative repair. Approaches include ranking candidates by test agreement, self-debugging from stack traces, and generating tests to expose faults. This grounds code models in an objective oracle, letting them fix functional bugs that pure next-token likelihood cannot detect.
UnknownDifficulty 6/10Verified
1782024Open-Weight Model Families
Technical report
Sébastien Bubeck, Marah Abdin, Dirk Groeneveld, Guilherme Penedo, Jie Tang, Gemma Team (Google DeepMind), Shanghai AI Laboratory
Gemma, Phi (small models trained on curated/synthetic 'textbook-quality' data), Falcon, Command R (retrieval and tool-use oriented), Yi, and GLM each shipped capable open-weight models with differing licenses and disclosure. OLMo is fully open: alongside weights it releases the Dolma training corpus, the training and evaluation code, and intermediate checkpoints, making it a scientific artifact rather than only a usable model. The distinction matters because most 'open' models share only weights, whereas OLMo lets researchers study and reproduce how a given capability arose during training.
Open weightDifficulty 5/10Verified
1792021Open-Weight Model Families
Peer reviewed
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Zhilin Yang, Jie Tang, Aohan Zeng
GLM masks spans of text and trains the model to regenerate them autoregressively while attending bidirectionally to the surrounding context, giving one model both the understanding strengths of masked encoders like BERT and the generation ability of decoder-only models. The team scaled the recipe to GLM-130B, a 130-billion-parameter bilingual model released with open weights and INT4 inference support so it could run on modest hardware. It anchored the GLM/ChatGLM family from Zhipu AI and Tsinghua and became one of the more capable early open bilingual foundation models.
Open weightDifficulty 5/10Verified
1802025Open-Weight Model Families
Technical report
Zhipu AI, Z.ai GLM Team, Jie Tang, Aohan Zeng
From GLM-4.5 (a 355B-parameter sparse mixture-of-experts model with 32B active, July 2025) through the GLM-5 family and the GLM-5.2 flagship (a roughly 750B-parameter MoE with sparse attention and a million-token context, 2026), Zhipu AI released the weights on Hugging Face under MIT, with FP8 variants for cheaper serving. The series targets reasoning and agentic software engineering and is also served through the z.ai API, but the checkpoints are genuinely public and self-hostable. It is among the most capable open-weight lines outside the Llama, Qwen, and DeepSeek families. (GLM-5.x release dates are approximate, pinned from repository timestamps rather than day-precise announcements.)
Open weightDifficulty 6/10Verified
1812022Open-Weight Model Families
Technical report
Teven Le Scao, Angela Fan, Christopher Akiki, Thomas Wolf, Stella Biderman
BLOOM was trained by the BigScience workshop on the ROOTS corpus spanning 46 natural and 13 programming languages, with the model, code, and data documentation released openly under a responsible-AI licence. It brought many languages underserved by English-centric models into a large open model and made the full training process transparent. It was an early proof that open, multi-institution collaboration could build models at frontier scale.
Open weightDifficulty 5/10Verified
1822023Inference & Serving
Software release
Georgi Gerganov, Lianmin Zheng, Ying Sheng, Ion Stoica, NVIDIA
llama.cpp plus the GGUF format made quantized models runnable on laptops and CPUs; TensorRT-LLM optimizes inference on NVIDIA GPUs; SGLang targets programs that make many related LLM calls. SGLang's RadixAttention keeps a radix tree of previously computed key-value cache so that requests sharing a prompt prefix (few-shot templates, agent loops, branching decodes) skip recomputation, and its language lets developers express constrained, multi-step generations. The result is substantially higher throughput for the structured, repetitive call patterns that agents and serving systems generate.
Source availableDifficulty 5/10Verified
1832020Long Context & Efficient Sequences
Peer reviewed
Iz Beltagy, Matthew E. Peters, Arman Cohan, Manzil Zaheer, Amr Ahmed
Full self-attention costs grow with the square of sequence length, capping practical context. Longformer uses sliding-window local attention plus a few global tokens; BigBird adds random connections and shows this local+global+random combination is a universal approximator and Turing-complete, so the sparsity does not sacrifice power. By making attention scale linearly, these models pushed usable context from around 512 into the thousands of tokens for tasks like long-document QA and genomics.
UnknownDifficulty 5/10Verified
1842024Long Context & Efficient Sequences
Peer reviewed
Hao Liu, Matei Zaharia, Pieter Abbeel, Tsendsuren Munkhdalai, Manaal Faruqui, DeepSeek-AI
Ring Attention splits a long sequence across multiple devices, and each device computes its local block of attention while key-value chunks circulate around the ring, overlapping communication with computation so the effective context grows with the number of devices without approximation. Infini-Attention instead keeps a bounded compressive memory that accumulates old key-value information, letting a fixed-size model attend to unbounded history. Multi-head Latent Attention (MLA, from DeepSeek) compresses the key-value cache into a low-rank latent to shrink the memory each token costs at inference. Together they attack long context from three angles: distribute it, compress the history, or compress the cache.
UnknownDifficulty 6/10Verified
1852023Long Context & Efficient Sequences
Peer reviewed
Shouyuan Chen, Yuandong Tian, Bowen Peng, Jeffrey Quesnelle, Enrico Shippole
Transformers trained with rotary position embeddings degrade sharply when run past their training context length because they encounter position rotations never seen in training. These methods rescale the RoPE frequencies so longer positions map back into the range the model already learned: Position Interpolation linearly compresses positions, NTK-aware scaling adjusts per-frequency to preserve high-frequency detail, and YaRN combines frequency-selective interpolation with an attention-temperature correction. The result is context windows extended by large factors (e.g. 4x-16x) after only brief fine-tuning or none at all, avoiding the cost of pretraining from scratch on long sequences.
UnknownDifficulty 5/10Verified
1862023Long Context & Efficient Sequences
Peer reviewed
Bo Peng, Michael Poli, Stefano Ermon, Christopher Ré, Opher Lieber, Yoav Shoham
Standard attention costs compute and memory that grow quadratically with sequence length and requires a KV cache that grows linearly, making long sequences expensive. This line of work reformulates sequence mixing as linear-cost operations: RWKV casts a transformer-like model as an RNN with constant per-token state, Hyena uses long implicit convolutions with gating, and hybrids like Jamba interleave state-space (Mamba) layers with a few attention layers. These architectures achieve near-linear scaling in sequence length and constant or bounded inference memory while remaining competitive with transformers on language modeling.
UnknownDifficulty 6/10Verified
1872023Long Context & Efficient Sequences
Peer reviewed
Nelson F. Liu, Kevin Lin, Percy Liang, Cheng-Ping Hsieh, Boris Ginsburg
The work tested models on multi-document QA and key-value retrieval while varying where the relevant information was placed in a long input. Accuracy followed a U-shaped curve: high when the needed fact was at the beginning or end, substantially lower when buried in the middle, even for models nominally supporting the full length. This demonstrated that advertised context length overstates usable context, motivating position-robustness work and synthetic stress benchmarks like RULER that measure at what effective length a model still retrieves and reasons reliably.
UnknownDifficulty 5/10Verified
1882023Evaluation & Benchmarks
Benchmark paper
David Rein, et al.
Existing knowledge benchmarks had largely saturated and were answerable by search, so they no longer separated genuine reasoning from retrieval. GPQA collects questions in biology, physics, and chemistry authored by PhD-level experts, then filters to items where other experts agree on the answer but skilled non-experts fail even after extended Googling (validated 'Google-proof'). This yields a small, high-difficulty set that measures whether a model can reason at graduate level and serves as a hard target and a testbed for scalable-oversight research.
UnknownDifficulty 4/10Verified
1892022Evaluation & Benchmarks
Benchmark paper
Aarohi Srivastava, Mirac Suzgun, Jason Wei, Percy Liang, Rishi Bommasani, Tony Lee
As models grew, single-benchmark scores gave a narrow and often misleading picture of capability. BIG-bench assembled hundreds of diverse community tasks (with BBH isolating the subset where models then lagged humans), and HELM evaluated many models on a common set of scenarios reporting not just accuracy but calibration, robustness, fairness, bias, toxicity, and efficiency side by side. Together they made evaluation multi-dimensional and comparable across models, revealing capability gaps and cost/quality trade-offs that a lone accuracy number hides.
UnknownDifficulty 5/10Verified
1902023Evaluation & Benchmarks
Benchmark paper
Xiao Liu, Jie Tang, Shuyan Zhou, Graham Neubig, Grégoire Mialon, Thomas Scialom, Shunyu Yao, Karthik Narasimhan
Static QA benchmarks cannot measure whether a model can plan and act over many steps against a stateful environment. WebArena provides self-hosted realistic websites where agents must navigate and complete tasks judged by end-state, AgentBench spans multiple environments (OS, database, web, games), GAIA poses real-world questions needing tool use and multi-hop reasoning, and tau-bench simulates tool-and-user customer-service dialogues. By scoring task completion in executable settings, they exposed a large gap between chat competence and reliable agentic execution.
UnknownDifficulty 5/10Verified
1912021Evaluation & Benchmarks
Benchmark paper
Stephanie Lin, Owain Evans, Colin White, Tom Goldstein, Jason Wei, William Fedus
Because static public benchmarks leak into training corpora, high scores can reflect memorization rather than capability. These efforts attack that differently: TruthfulQA targets questions where imitating human text produces confident falsehoods, LiveBench continuously draws fresh questions from recent sources (papers, news, competitions) with objective ground truth and rotates them out to limit contamination, and SWE-bench-verified is a human-filtered subset of real GitHub issues confirmed to be well-specified and solvable. The shared idea is to keep evaluation valid over time by making the test set hard to have already seen or trivially matched.
UnknownDifficulty 4/10Verified
1922024Scaling Laws & Compute
Vision essay
Leopold Aschenbrenner
The long essay projects the straight lines on graphs of training compute and efficiency forward, argues that the gap from current models to human-level research assistants is a few more orders of magnitude, and discusses security and governance consequences. It is a prediction and advocacy piece, not an empirical result, and its timelines are contested. It captured a strand of frontier-lab thinking in 2024.
UnknownDifficulty 3/10Verified
1932018Scaling Laws & Compute
Technical report
Dario Amodei, Danny Hernandez
The analysis charts training compute across landmark systems and finds a doubling time far faster than Moore’s law over the studied period. It made the compute-growth trend legible and widely cited as evidence that scaling, not just architecture, was driving progress. It is an early empirical anchor for the scaling narrative.
UnknownDifficulty 3/10Verified
1942021Agents & Tool Use
Preprint
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, John Schulman
WebGPT gives GPT-3 a text-based browser and trains it with human comparisons to search, navigate, and quote evidence, then optimizes against a learned reward model. The result answers long-form questions with references that raters prefer, at the cost of sometimes over-trusting sources. It was an early demonstration of tool use and retrieval-augmented answering with RLHF.
UnknownDifficulty 6/10Verified
1952022Agents & Tool Use
Preprint
Michael Ahn, Anthony Brohan, Noah Brown, Karol Hausman
SayCan scores candidate skills by multiplying an LLM’s estimate that a skill helps achieve the instruction with a value function’s estimate that the skill is currently feasible, so the robot picks useful and possible actions. This let a robot carry out long natural-language instructions in a kitchen. It was an influential early bridge from language models to embodied action.
UnknownDifficulty 6/10Verified
1962022Reasoning & Test-Time Compute
Preprint
Aitor Lewkowycz, Anders Andreassen, David Dohan, Vinay Ramasesh
Minerva continues pretraining a large language model on math-heavy web and arXiv data, then uses step-by-step prompting and samples many solutions to vote on the answer. It substantially raised scores on MATH and STEM problem sets using only the model’s own reasoning. It marked how far scaled models plus test-time sampling could push mathematical reasoning.
UnknownDifficulty 6/10Verified
1972022Multimodality
Preprint
Chitwan Saharia, William Chan, Saurabh Saxena, Mohammad Norouzi
Imagen pairs a frozen T5-XXL language-model text encoder with a cascade of diffusion models and finds that scaling the text encoder improves image-text alignment more than scaling the image model. It reached strong photorealism and prompt fidelity on standard evaluations. It clarified the role of language understanding in text-to-image systems.
UnknownDifficulty 6/10Verified
1982024Multimodality
Technical report
Tim Brooks, Bill Peebles, OpenAI
Sora represents videos as spacetime patches and trains a diffusion Transformer over them, scaling data and compute rather than baking in physics. The technical report shows long, high-resolution, temporally consistent clips and argues that scale yields rudimentary simulation of objects and dynamics. It signaled diffusion-Transformer scaling as a path to video and world models.
UnknownDifficulty 5/10Verified
1992025Continual Learning & Memory
Preprint
Ali Behrouz, Peilin Zhong, Vahab Mirrokni
Titans pairs short-term attention with a deep neural memory updated online by a surprise-based gradient signal, so salient new information is written into the memory as the model reads, with selective forgetting. This yields effective context well beyond the attention window while keeping inference cost bounded. It is a leading example of the test-time-learning branch, where memory lives in parameters updated at inference rather than only in the context window.
UnknownDifficulty 6/10Verified
2001997Neural Foundations
Peer reviewed
Mike Schuster, Kuldip Paliwal
The architecture processes a sequence left-to-right with one recurrent network and right-to-left with a second, then merges the two hidden states at each position before making a prediction. This gives every output access to the entire input sequence rather than only the elements seen so far. Because it requires the whole sequence up front it suits offline labeling tasks like speech and text tagging, and the bidirectional design later combined with LSTMs to become a standard component for sequence labeling.
UnknownDifficulty 4/10Verified
2012006Neural Foundations
Peer reviewed
Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber
CTC adds a special 'blank' symbol and defines the probability of a target label sequence as the sum over all frame-level paths that collapse to it (by removing blanks and merging repeats), computed efficiently with a forward-backward dynamic-programming algorithm. This lets the network output a probability distribution over labels at every time step and be trained end-to-end on unsegmented pairs of input and target sequences. It removed the requirement for hand-aligned data and enabled direct recurrent-network sequence transcription in speech and handwriting recognition.
UnknownDifficulty 5/10Verified
2022019Encoders
Peer reviewed
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut
ALBERT decouples the vocabulary embedding size from the hidden size (factorizing that large matrix) and shares the same parameters across all Transformer layers, drastically reducing the model's parameter footprint. It also replaces BERT's next-sentence prediction with a sentence-order prediction task that forces the model to learn inter-sentence coherence rather than topic overlap. Together these let ALBERT scale hidden dimensions and depth without a proportional memory blowup, reaching stronger results than BERT-large at a fraction of the stored parameters.
UnknownDifficulty 5/10Verified
2032023Encoder–Decoders
Peer reviewed
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, et al.
UL2 trains on a mixture of denoising objectives (short spans, long spans, and sequential/prefix-LM corruption) and prepends a mode token that tells the model which denoising regime applies, letting it be switched between modes at inference. This bridges the gap where masked-denoising models excelled at fine-tuning while causal LMs excelled at few-shot prompting, giving one recipe that performs on both. The framework is architecture-agnostic and let a single pretrained model be adapted to supervised fine-tuning and prompting without choosing an objective in advance.
UnknownDifficulty 6/10Verified
2042018Data, Corpora & Tokenization
Peer reviewed
Taku Kudo
Standard subword methods like BPE give one deterministic segmentation per word, so the model never sees alternative splits. This work defines a unigram LM over subwords that can yield multiple probable segmentations and samples among them each epoch, exposing the model to varied tokenizations of identical text. Acting as data augmentation, it improved neural machine translation accuracy especially on low-resource and noisy settings, and the unigram tokenizer became a widely used alternative to BPE (shipped in SentencePiece).
UnknownDifficulty 5/10Verified
2052023Data, Corpora & Tokenization
Peer reviewed
Sang Michael Xie, Hieu Pham, Xuanyi Dong, et al.
The method first trains a small reference model, then trains a second small proxy model with Group Distributionally Robust Optimization that raises the sampling weight of domains where the proxy has the largest excess loss over the reference. The resulting domain weights are reused to sample data for a much larger model. This removed the need to hand-tune or grid-search corpus mixtures at full scale, and reached target accuracy in fewer training steps by reweighting domains like The Pile rather than using their default proportions.
UnknownDifficulty 6/10Verified
2062019Distributed Training
Peer reviewed
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, et al.
Rather than draining the pipeline between minibatches, PipeDream uses a one-forward-one-backward schedule that overlaps the forward and backward work of different minibatches to keep all stages busy. Because a minibatch's backward pass then sees newer weights than its forward pass, it stashes the weight version used in the forward pass and reuses it in the backward pass so gradients stay consistent. An automatic profiler partitions layers across stages to balance compute and communication, cutting time-to-target-accuracy relative to data-parallel and flush-based pipeline training.
UnknownDifficulty 6/10Verified
2072023Reasoning & Test-Time Compute
Peer reviewed
Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman
By inserting biasing features into prompts, such as always marking option (A) as correct, the authors showed models would follow the bias while writing plausible reasoning that never mentioned it, demonstrating that CoT explanations can be post-hoc rather than causal. Related findings cover overthinking, where longer reasoning hurts accuracy, and contamination, where memorized answers masquerade as derived ones. The collective correction is that a readable chain-of-thought is not automatically a faithful account of the computation, so it cannot be trusted as an interpretability or safety signal without further verification.
UnknownDifficulty 6/10Verified
2082023Agents & Tool Use
Preprint
Charles Packer, Joseph E. Gonzalez, Qingyun Wu, Chi Wang, Guohao Li, Bernard Ghanem
MemGPT borrows virtual-memory ideas, treating the context window as limited RAM and external storage as disk, so the agent pages information in and out under its own control to maintain long-running conversations and documents beyond the token limit. AutoGen provides a framework for defining multiple conversable agents that message each other and call tools to solve a task jointly, and CAMEL uses role-playing prompts to make two agents cooperate through dialogue with minimal human steering. The shared contribution is infrastructure for memory that outlives a single context and for coordinating several agents, enabling longer and more complex workflows than a single stateless call.
UnknownDifficulty 6/10Verified
2092024Retrieval & Memory
Technical report
Darren Edge, Ha Trinh, et al.
GraphRAG uses a language model to extract entities and relationships into a graph, clusters the graph into communities, and precomputes summaries for each community; queries are then answered by combining these community summaries rather than only retrieving isolated text chunks. This structured approach lets the system answer broad, whole-corpus sensemaking questions (such as overarching themes) that flat vector retrieval handles poorly, while keeping evidence traceable to source entities and relations.
UnknownDifficulty 5/10Verified
2102020Long Context & Efficient Sequences
Peer reviewed
Nikita Kitaev, Łukasz Kaiser, Krzysztof Choromanski, Adrian Weller, Angelos Katharopoulos, François Fleuret
Softmax attention requires materializing an N-by-N similarity matrix; these methods avoid it. Performer's FAVOR+ maps queries and keys through random feature functions whose dot products approximate the softmax kernel unbiasedly, letting attention be reassociated into linear-time operations. Reformer used locality-sensitive hashing and reversible layers to similar ends, and Linear Transformers recast attention as a kernelized RNN. This traded exact attention for near-linear scaling, enabling much longer sequences on fixed memory, though with some approximation cost that limited adoption at frontier scale.
UnknownDifficulty 6/10Verified
2112023Agents & Tool Use
Software release
Toran Bruce Richards, Yohei Nakajima
Given a high-level goal, these programs prompt the model to generate a task list, execute tasks one at a time using tools like web search and file access, and feed results back to reprioritize and spawn new tasks, running with little human intervention. BabyAGI centered on a compact task-creation and prioritization loop backed by a vector store, while AutoGPT added tool integrations and persistence around a similar cycle. They were engineering demonstrations rather than research papers, and their viral spread popularized the autonomous-agent pattern while also exposing its practical limits in reliability, looping, and cost.
Source availableDifficulty 3/10Verified