Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

38 records

0011986Neural Foundations

Foundational paper

Backpropagation (Learning representations by back-propagating errors)

David Rumelhart, Geoffrey Hinton, Ronald Williams

The paper applies the chain rule to compute, for every weight in a layered network, how much it contributed to the output error, then adjusts weights in the direction that reduces that error. This gave hidden units a way to learn useful internal representations instead of being hand-designed. It made training networks with one or more hidden layers a routine gradient-descent procedure and became the standard method for supervised neural network learning.

UnknownDifficulty 5/10Verified
Read record Primary source
0022014Neural Foundations

Peer reviewed

Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau attention)

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

Instead of forcing the encoder to compress a whole sentence into one vector, this model learns an alignment: for each output word the decoder scores and softly selects relevant source positions to build a context vector. This kept translation quality from degrading as sentences got longer and produced interpretable soft alignments between source and target words. The mechanism became the core building block that the Transformer later generalized.

UnknownDifficulty 5/10Verified
Read record Primary source
0032015Neural Foundations

Peer reviewed

Deep Residual Learning for Image Recognition (ResNet)

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

The authors added shortcut connections that add a layer block's input to its output, so each block only has to learn the difference from identity. This made gradients flow through very deep stacks and allowed networks of 50 to 152 layers to train and improve rather than degrade, winning ImageNet 2015. Residual blocks became a default component in most later deep vision and language architectures.

UnknownDifficulty 4/10Verified
Read record Primary source
0042014Neural Foundations

Peer reviewed

Adam: A Method for Stochastic Optimization

Diederik Kingma, Jimmy Ba

Adam keeps exponentially decaying averages of past gradients and squared gradients, applies bias correction to those estimates, and uses them to scale each parameter's step. This gives well-behaved step sizes across parameters with different gradient magnitudes and works with little tuning across many problems. Its combination of low memory cost and robust default hyperparameters made it the default optimizer for training deep networks.

UnknownDifficulty 4/10Verified
Read record Primary source
0052014Neural Foundations

Peer reviewed

Sequence to Sequence Learning with Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc Le

The paper used one multilayer LSTM to read an entire source sentence into a single vector and a second LSTM to generate the target sentence from that vector. A key practical trick was reversing the order of source words, which shortened the average distance between corresponding input and output tokens and made optimization easier, pushing WMT'14 English-to-French BLEU to competitive levels. It showed that a purely neural, general-purpose sequence model could rival phrase-based statistical MT and became the template for later encoder-decoder systems.

UnknownDifficulty 5/10Verified
Read record Primary source
0062016Neural Foundations

Preprint

Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey Hinton

The method computes each layer's normalization statistics over the units within one example, so behavior is identical during training and testing and does not depend on batch composition. This made it effective for RNNs and sequence models where batch normalization worked poorly, stabilizing and speeding up training. Layer normalization became standard in Transformer architectures.

UnknownDifficulty 4/10Verified
Read record Primary source
0072015Neural Foundations

Peer reviewed

Neural Machine Translation of Rare Words with Subword Units (BPE)

Rico Sennrich, Barry Haddow, Alexandra Birch

BPE starts from characters and iteratively merges the most frequent adjacent symbol pairs to build a vocabulary of subword units, so any word can be encoded as a sequence of known pieces. This let translation models handle rare words, compounds, and morphology without a huge word vocabulary or a separate back-off mechanism. Subword segmentation via BPE became standard tokenization for most later NMT and language models.

UnknownDifficulty 4/10Verified
Read record Primary source
0082012Neural Foundations

Peer reviewed

AlexNet: ImageNet Classification with Deep CNNs

Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton

AlexNet is a deep convolutional network with roughly 60 million parameters trained on over a million labeled images across 1,000 categories, using ReLU activations for faster training, dropout to limit overfitting, data augmentation, and two GPUs to fit the model in memory. Combining these let it cut the ImageNet top-5 error far below the previous best. Its result shifted computer vision toward deep learning and drove wide adoption of GPU-trained convolutional networks.

UnknownDifficulty 4/10Verified
Read record Primary source
0091997Neural Foundations

Peer reviewed

Long Short-Term Memory (LSTM)

Sepp Hochreiter, Jürgen Schmidhuber, Felix Gers

Each LSTM unit keeps a memory cell whose value is protected by gates that control when new information is written in, retained, or read out, so the cell can hold information across many time steps. Because the cell passes its state forward with near-constant behavior when the gates allow, gradients do not shrink away over long sequences the way they do in plain recurrent networks. This let recurrent networks learn dependencies spanning hundreds of steps and made them effective for sequence tasks such as speech and language modeling.

UnknownDifficulty 5/10Verified
Read record Primary source
0102009Neural Foundations

Peer reviewed

ImageNet: A Large-Scale Hierarchical Image Database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei

ImageNet organised millions of human-labelled images into thousands of categories following the WordNet hierarchy, assembled through large-scale crowdsourcing. It supplied the data and the annual ILSVRC challenge that made it possible to train and fairly compare very large vision models. AlexNets 2012 win on this benchmark is widely taken as the moment deep learning became the dominant approach in vision.

UnknownDifficulty 3/10Verified
Read record Primary source
0112003Neural Foundations

Peer reviewed

A Neural Probabilistic Language Model

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin

Instead of treating words as discrete symbols, the model maps each word to a real-valued vector and feeds the vectors of the preceding words into a neural network that outputs a probability distribution over the next word. Words used in similar contexts end up with similar vectors, so the model can assign reasonable probability to word sequences it never saw in training. This showed that a single network could learn word features and the probability function together, and set the template later used by neural word embeddings and language models.

UnknownDifficulty 5/10Verified
Read record Primary source
0122013Neural Foundations

Preprint

Efficient Estimation of Word Representations (Word2Vec)

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

The models drop the hidden nonlinear layers of earlier neural language models and instead train a shallow network to predict a word from its surrounding context (CBOW) or the surrounding context from a word (skip-gram). This makes training fast enough to run on billions of words, producing vectors where semantic and syntactic relationships appear as consistent offsets (the vector arithmetic 'king minus man plus woman' lands near 'queen'). It made high-quality word embeddings cheap to compute and widely reusable as input features for other NLP systems.

UnknownDifficulty 4/10Verified
Read record Primary source
0132014Neural Foundations

Peer reviewed

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov

During training, each unit is kept with some probability and dropped otherwise, so the network cannot rely on any specific combination of units and must learn features that are useful on their own. At test time all units are used with their outputs scaled to match, which approximates averaging over the many thinned networks seen during training. This reduced overfitting on a range of vision and speech tasks and became a standard technique for training large networks with limited data.

UnknownDifficulty 3/10Verified
Read record Primary source
0142010Neural Foundations

Peer reviewed

Understanding the Difficulty of Training Deep Nets (Xavier/He init)

Xavier Glorot, Yoshua Bengio, Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

The paper measures how the variance of activations and back-propagated gradients changes from layer to layer under different setups, showing that poor scaling makes deep networks with saturating activations (like sigmoid) hard to train. It proposes initializing weights with a variance set from the number of incoming and outgoing connections so that signals neither vanish nor blow up as they pass through the layers. This initialization let deeper networks converge more reliably and became a default starting point for many architectures.

UnknownDifficulty 5/10Verified
Read record Primary source
0152010Neural Foundations

Peer reviewed

Rectified Linear Units (ReLU)

Vinod Nair, Geoffrey Hinton, Xavier Glorot, Antoine Bordes, Yoshua Bengio

A ReLU passes positive inputs through unchanged and outputs zero for negative inputs, so its gradient is a constant one wherever the unit is active instead of shrinking toward zero the way sigmoid and tanh gradients do. This keeps gradients flowing through many layers and produces sparse activations that are cheap to compute. Using ReLUs let deep networks train faster and reach lower error, and it became the default activation for most feedforward and convolutional networks.

UnknownDifficulty 3/10Verified
Read record Primary source
0162015Neural Foundations

Peer reviewed

Batch Normalization

Sergey Ioffe, Christian Szegedy

For each mini-batch, the method normalizes a layer's pre-activation values to zero mean and unit variance, then applies a learned scale and shift so the layer can still represent what it needs. This keeps the distribution of inputs to each layer more stable as the weights below it change, which allows much higher learning rates and less sensitivity to initialization. It sped up training substantially, added a mild regularizing effect, and became a standard component of deep convolutional networks.

UnknownDifficulty 4/10Verified
Read record Primary source
0171998Neural Foundations

Peer reviewed

LeNet: Gradient-Based Learning Applied to Document Recognition

Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner

The network stacks convolutional layers that apply the same small learned filters across the image with pooling layers that reduce resolution, so it detects local patterns regardless of position while keeping the parameter count low through weight sharing. Trained on labeled digit images, it learns its own feature detectors instead of relying on manually designed ones. It achieved strong accuracy on handwritten digit recognition and was deployed to read amounts on bank checks, demonstrating a practical gradient-trained vision system.

UnknownDifficulty 4/10Verified
Read record Primary source
0182006Neural Foundations

Peer reviewed

Deep Belief Nets / Reducing Dimensionality

Geoffrey Hinton, Simon Osindero, Yee-Whye Teh, Ruslan Salakhutdinov

The method builds a deep belief network by training each layer in turn to model the activations of the layer below, giving the full network a sensible starting point before ordinary gradient fine-tuning. Applied to an autoencoder, this two-stage approach learns a compact code that reconstructs the input better than linear methods like PCA. The work provided a practical way to train deep networks before better initialization and activation functions existed, and helped renew interest in deep learning.

UnknownDifficulty 5/10Verified
Read record Primary source
0191988Neural Foundations

Peer reviewed

Learning to Predict by the Methods of Temporal Differences

Richard S. Sutton

Sutton showed how to learn multi-step predictions incrementally: instead of comparing a prediction only to the eventual outcome, TD updates each prediction using the difference between successive predictions along the way. This let learning happen online, step by step, and made better use of experience than earlier supervised methods. TD became the computational core of most later reinforcement learning, including value estimation in games and control.

UnknownDifficulty 6/10Verified
Read record Primary source
0201992Neural Foundations

Peer reviewed

Q-learning

Christopher J. C. H. Watkins, Peter Dayan

Watkins and Dayan defined a rule that updates an estimate of the long-run reward for each state-action pair (the Q-value) from observed transitions and rewards. Because updates use the best available next action rather than the action actually taken, the agent can learn the optimal policy while still exploring. They gave a convergence proof, giving reinforcement learning a model-free method with mathematical guarantees that underpins much later work.

UnknownDifficulty 5/10Verified
Read record Primary source
0211992Neural Foundations

Peer reviewed

Simple Statistical Gradient-Following Algorithms (REINFORCE)

Ronald J. Williams

Williams derived how to nudge the weights of a stochastic network so that actions leading to higher reward become more likely, using only the reward signal and the probability of the chosen action. The estimate of the gradient is unbiased and needs no model of the environment. This policy-gradient approach became the basis for later actor-critic and deep policy-optimization methods used to train agents and language models.

UnknownDifficulty 6/10Verified
Read record Primary source
0222015Neural Foundations

Peer reviewed

Human-level Control through Deep RL (DQN)

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al.

Mnih and colleagues trained a network to estimate action values from screen images, using two stabilizing tricks: an experience replay buffer that reuses and decorrelates past transitions, and a periodically updated target network. The same model and hyperparameters learned to play 49 Atari games from pixels and reward alone. It showed that deep networks could serve as reliable function approximators in reinforcement learning, launching the deep RL field.

UnknownDifficulty 6/10Verified
Read record Primary source
0232016Neural Foundations

Peer reviewed

Mastering the Game of Go (AlphaGo/AlphaZero)

David Silver, Aja Huang, Chris J. Maddison, Julian Schrittwieser, Thomas Hubert, Demis Hassabis

AlphaGo paired a policy network (which moves to consider) and a value network (who is winning) with tree search to defeat top human Go players. AlphaZero simplified this into a single network trained purely by playing against itself, guided by search, and generalized to chess and shogi. The work showed that self-play plus search could reach superhuman strength without human examples or handcrafted evaluation.

UnknownDifficulty 6/10Verified
Read record Primary source
0242013Neural Foundations

Peer reviewed

Auto-Encoding Variational Bayes (VAE)

Diederik P. Kingma, Max Welling

Kingma and Welling framed generation as learning a latent variable model and approximating its intractable posterior with a neural encoder. Their key move, the reparameterization trick, rewrote random sampling so gradients could flow through it, allowing the encoder and decoder to be trained together by ordinary backpropagation on a single objective (the evidence lower bound). This gave a scalable way to learn continuous latent representations and generate new data, widely used for images and representation learning.

UnknownDifficulty 6/10Verified
Read record Primary source
0252014Neural Foundations

Peer reviewed

Generative Adversarial Networks (GAN)

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al.

Goodfellow and colleagues set up a game between two networks: a generator that turns noise into samples and a discriminator that tries to tell real data from generated data. Training the two in competition drives the generator to match the real data distribution without needing an explicit likelihood. This adversarial framework produced sharp, realistic image generation and spawned a large family of follow-on models for synthesis and translation.

UnknownDifficulty 6/10Verified
Read record Primary source
0262014Neural Foundations

Peer reviewed

GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher Manning

GloVe builds a matrix counting how often each word appears near each other word across the whole corpus, then trains word vectors so that the dot product of two vectors approximates the logarithm of their co-occurrence count, weighting frequent pairs so common words do not dominate. This uses aggregate statistics directly rather than scanning context windows one at a time like word2vec. The resulting vectors placed related words near each other and encoded analogies as vector offsets, and became a widely used pretrained input for later NLP models.

UnknownDifficulty 4/10Verified
Read record Primary source
0272015Neural Foundations

Peer reviewed

Effective Approaches to Attention-based NMT (Luong attention)

Minh-Thang Luong, Hieu Pham, Christopher Manning

The paper lets a translation decoder, at each output step, compute a weighted average of encoder hidden states where the weights come from a similarity score between the current decoder state and each source state, then combines that context with the decoder state to predict the next word. It compared scoring functions (dot product, general, concat) and introduced a 'local' attention that only looks at a small window of source positions to save computation. These variants improved translation quality and clarified the design space, and the dot-product scoring it favored became standard in later attention and Transformer work.

UnknownDifficulty 5/10Verified
Read record Primary source
0281989Neural Foundations

Peer reviewed

Universal Approximation Theorem

George Cybenko, Kurt Hornik

Cybenko (1989) showed this for sigmoid-type activations and Hornik (1991) generalized it to broad classes of activation functions, proving that finite sums of scaled, shifted activations are dense in the space of continuous functions. The result is an existence guarantee: for any target accuracy some network with enough hidden units achieves it. It does not say how many units are needed or how to find the weights by training, but it removed the theoretical objection that networks might be fundamentally incapable of representing complex functions.

UnknownDifficulty 5/10Verified
Read record Primary source
0291990Neural Foundations

Peer reviewed

Finding Structure in Time (Elman RNN)

Jeffrey Elman

The network copies its hidden-layer state at each step into a 'context' layer that is fed back in alongside the next input, so the hidden units learn representations that depend on prior elements of the sequence. Trained on tasks like predicting the next word or discovering word boundaries in a stream, it developed internal representations that reflected grammatical categories and structure without being told them. This showed that temporal structure could be handled by recurrence rather than by explicit time-delay windows, and the architecture became a foundation for later recurrent networks.

UnknownDifficulty 4/10Verified
Read record Primary source
0301994Neural Foundations

Peer reviewed

Learning Long-Term Dependencies is Difficult (vanishing gradients)

Sepp Hochreiter, Yoshua Bengio, Patrice Simard, Paolo Frasconi

The paper showed mathematically that for a recurrent network to store information robustly over time its dynamics must contract, but contraction causes error gradients passed backward through many time steps to shrink toward zero, so distant past inputs contribute almost nothing to weight updates. When the dynamics instead expand, gradients blow up. This trade-off means standard gradient descent cannot reliably link causes and effects separated by many steps. The analysis motivated later remedies such as gating (LSTM, GRU), gradient clipping, and better initialization.

UnknownDifficulty 5/10Verified
Read record Primary source
0312014Neural Foundations

Peer reviewed

Gated Recurrent Unit (GRU) / RNN Encoder-Decoder

Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, Yoshua Bengio

One recurrent network (the encoder) reads a source sentence and compresses it into a fixed-length vector, and a second recurrent network (the decoder) generates the target sentence from that vector, trained jointly to maximize the probability of the correct output. To make the recurrence trainable over longer spans the paper introduced the GRU, which uses update and reset gates to control how much past state is kept or overwritten, achieving LSTM-like behavior with a simpler structure. The encoder-decoder setup became the template for neural machine translation and sequence-to-sequence learning, and its learned phrase representations improved a statistical translation system.

UnknownDifficulty 4/10Verified
Read record Primary source
0322011Neural Foundations

Peer reviewed

Natural Language Processing (almost) from Scratch

Ronan Collobert, Jason Weston, Léon Bottou, et al.

The system maps words to learned vector embeddings, processes sentences with a convolutional network, and is trained jointly across tasks like part-of-speech tagging, chunking, named-entity recognition, and semantic role labeling. Crucially it pretrains word embeddings on large unlabeled corpora using a ranking objective, then reuses them, so the model discovers useful features rather than relying on task-specific hand-crafted inputs. It reached competitive accuracy across tasks with one architecture, demonstrating the transfer value of unsupervised pretrained word representations that later work built on.

UnknownDifficulty 5/10Verified
Read record Primary source
0331993Neural Foundations

Peer reviewed

The Mathematics of Statistical MT (IBM alignment models)

Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, Robert L. Mercer

The paper frames translation as finding the target sentence most probable given the source, decomposed via Bayes' rule into a translation model and a language model, and introduces a hidden alignment variable saying which source word generated each target word. The five IBM models add successive detail (word translation probabilities, then absolute position, fertility of how many words a source word produces, and distortion of position), and their parameters are estimated from unaligned sentence pairs using the EM algorithm. These alignment models became the statistical foundation of machine translation for roughly two decades and the basis for later phrase-based systems.

UnknownDifficulty 6/10Verified
Read record Primary source
0341995Neural Foundations

Peer reviewed

TD-Gammon

Gerald Tesauro

The program uses a neural network to estimate the expected outcome of a board position and trains it with TD(lambda), which adjusts predictions so that each position's estimated value moves toward the value of positions that follow it during play. By playing hundreds of thousands of games against itself it improved without expert-labeled positions or a hand-tuned evaluation function. It reached strong tournament-level play and even influenced human opening theory, providing an early demonstration that reinforcement learning combined with function approximation could master a large-state-space game.

UnknownDifficulty 5/10Verified
Read record Primary source
0352014Neural Foundations

Preprint

Neural Turing Machines / Memory Networks

Alex Graves, Greg Wayne, Ivo Danihelka, Jason Weston, Sumit Chopra, Antoine Bordes

A controller network emits read and write operations addressed to memory locations using soft, differentiable attention weights (based on content similarity and location shifting), so gradients can flow through the memory access and the model learns how to use its memory end-to-end. This separates computation from an addressable storage bank, letting the network learn procedures like copying, sorting, and associative recall and generalize them to longer inputs than seen in training. It, alongside Memory Networks, showed that networks augmented with external memory could learn algorithm-like behavior, influencing later memory-augmented and attention-based models.

UnknownDifficulty 6/10Verified
Read record Primary source
0362012Neural Foundations

Peer reviewed

Deep Neural Networks for Acoustic Modeling in Speech

Geoffrey Hinton, Li Deng, Dong Yu, et al.

In a speech recognizer the acoustic model estimates how likely each short audio frame corresponds to each sub-phonetic state; the paper describes replacing the standard Gaussian-mixture estimator with a deep neural network that takes several frames of audio features and outputs those state probabilities, often pretrained layer by layer and then fine-tuned. Reported jointly by teams at Microsoft, Google, IBM, and the University of Toronto, the deep networks lowered word error rates on multiple large-vocabulary benchmarks. The shared results gave the field consistent evidence that deep acoustic models were a general improvement, and they were adopted in commercial speech systems.

UnknownDifficulty 5/10Verified
Read record Primary source
0371997Neural Foundations

Peer reviewed

Bidirectional Recurrent Neural Networks

Mike Schuster, Kuldip Paliwal

The architecture processes a sequence left-to-right with one recurrent network and right-to-left with a second, then merges the two hidden states at each position before making a prediction. This gives every output access to the entire input sequence rather than only the elements seen so far. Because it requires the whole sequence up front it suits offline labeling tasks like speech and text tagging, and the bidirectional design later combined with LSTMs to become a standard component for sequence labeling.

UnknownDifficulty 4/10Verified
Read record Primary source
0382006Neural Foundations

Peer reviewed

Connectionist Temporal Classification (CTC)

Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber

CTC adds a special 'blank' symbol and defines the probability of a target label sequence as the sum over all frame-level paths that collapse to it (by removing blanks and merging repeats), computed efficiently with a forward-backward dynamic-programming algorithm. This lets the network output a probability distribution over labels at every time step and be trained end-to-end on unsegmented pairs of input and target sequences. It removed the requirement for hand-aligned data and enabled direct recurrent-network sequence transcription in speech and handwriting recognition.

UnknownDifficulty 5/10Verified
Read record Primary source