Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

16 records

0012021Multimodality

Peer reviewed

CLIP: Learning Transferable Visual Models from NL Supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, et al.

Two encoders (one for images, one for text) are trained so that a photo and its matching caption land close together in a shared embedding space while mismatched pairs are pushed apart. To classify a new image with no task-specific training, you write candidate labels as text prompts and pick the one whose embedding is closest to the image. Because the labels are open-ended text rather than a fixed category list, one model transfers to many recognition tasks and became a reusable component for retrieval and for later image-generation and vision-language systems.

UnknownDifficulty 5/10Verified
Read record Primary source
0022021Multimodality

Peer reviewed

Vision Transformer (An Image is Worth 16x16 Words)

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al.

The model splits an image into a grid of small patches (e.g. 16x16 pixels), linearly embeds each patch as if it were a word token, adds position embeddings, and feeds the sequence into a standard Transformer encoder. With no convolutions, it learns spatial structure from data alone, which works well only when pre-trained on very large image datasets. This let the same architecture used for language be reused for vision and scaled up with more data and compute rather than hand-designed vision components.

UnknownDifficulty 5/10Verified
Read record Primary source
0032021Multimodality

Peer reviewed

Highly Accurate Protein Structure Prediction with AlphaFold

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Demis Hassabis

AlphaFold pairs an attention-based Evoformer that reasons jointly over multiple-sequence alignments and residue-pair representations with a structure module that directly predicts atomic coordinates, trained end-to-end with protein geometry built in. At the CASP14 assessment it predicted many structures to within experimental error, a step change over prior methods. DeepMind released the code and a public database of predicted structures, reshaping computational structural biology.

UnknownDifficulty 7/10Verified
Read record Primary source
0042022Multimodality

Peer reviewed

Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al.

Frozen vision and language backbones are bridged by a Perceiver-style resampler that turns variable numbers of image features into a fixed set of tokens, plus gated cross-attention layers inserted into the language model so text generation can attend to images. Training on large interleaved image-text web data teaches the bridge without retraining the expensive backbones. This allowed few-shot performance on captioning and visual question answering by showing examples in the prompt, rather than fine-tuning a separate model per task.

UnknownDifficulty 6/10Verified
Read record Primary source
0052023Multimodality

Peer reviewed

LLaVA: Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

Image features from a frozen CLIP encoder are mapped into the language model's token space by a single trainable projection, so the model can treat an image as extra context alongside a text prompt. The team used a text-only GPT model to generate multimodal instruction-following examples (questions, descriptions, and reasoning about images) and fine-tuned on them. This showed that a capable image-chat assistant could be built cheaply from existing open components plus synthetic instruction data, and it became a common baseline architecture for open multimodal models.

UnknownDifficulty 4/10Verified
Read record Primary source
0062022Multimodality

Peer reviewed

Latent Diffusion (Stable Diffusion) / Diffusion Transformers (DiT)

Robin Rombach, Patrick Esser, Björn Ommer, William Peebles, Saining Xie

An autoencoder first compresses images into a smaller latent representation, and the denoising diffusion model learns to generate in that latent space rather than over millions of raw pixels. A cross-attention mechanism injects conditioning such as text prompts, and the decoder turns the final latent back into a full-resolution image. Working in the smaller space made training and inference cheap enough to run on consumer GPUs, which enabled the open release of a general-purpose text-to-image model.

UnknownDifficulty 6/10Verified
Read record Primary source
0072024Multimodality

System card

Native/Omni Multimodality (Gemini / GPT-4V / GPT-4o)

Google DeepMind, OpenAI

The report describes models built to accept and reason over mixed inputs (text, images, audio, and video) natively and to produce text and image outputs, trained jointly across modalities and offered in sizes from on-device to datacenter scale. Native multimodal training is meant to avoid the information loss of connecting independently trained encoders after the fact. The report reports benchmark results across language, coding, reasoning, and multimodal tasks; the underlying weights and full training details are not public, and related closed systems like GPT-4V and GPT-4o pursue similar goals.

UnknownDifficulty 7/10Verified
Read record Primary source
0082022Multimodality

Dataset paper

LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ludwig Schmidt

LAION-5B is a filtered, CLIP-scored collection of billions of image-caption pairs scraped from the web and released openly by the LAION non-profit. It supplied the training data behind Stable Diffusion and many open CLIP and vision-language models, shifting a major bottleneck -- data -- from proprietary to public. It also foregrounded the governance, consent, and safety questions that come with web-scale datasets.

UnknownDifficulty 4/10Verified
Read record Primary source
0092021Multimodality

Preprint

Zero-Shot Text-to-Image Generation (DALL-E)

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray

DALL-E encodes images as discrete tokens and trains a large Transformer to model the sequence of text followed by image tokens, then samples images from captions and reranks them with CLIP. It produced novel, compositional images without task-specific training. It opened the modern text-to-image line later dominated by diffusion methods.

UnknownDifficulty 6/10Verified
Read record Primary source
0102022Multimodality

Preprint

Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman

Whisper is an encoder-decoder Transformer trained on hundreds of thousands of hours of diverse audio-text pairs scraped from the web, covering many languages and tasks in one model. It approaches specialized systems zero-shot and is resilient to accents and noise. Its open release made strong speech recognition broadly available.

UnknownDifficulty 5/10Verified
Read record Primary source
0112021Multimodality

Peer reviewed

ALIGN: Scaling Up Visual and Vision-Language Representation Learning

Chao Jia, Yinfei Yang, Ye Xia, et al.

ALIGN trains separate image and text encoders to map matched image-caption pairs close together in a shared embedding space using a contrastive loss, learned over ~1.8 billion raw web image/alt-text pairs with only minimal frequency-based cleaning. By tolerating noisy supervision at scale, it removed the bottleneck of building large curated vision-language datasets and enabled strong zero-shot classification and image-text retrieval from web data alone.

UnknownDifficulty 5/10Verified
Read record Primary source
0122023Multimodality

Peer reviewed

BLIP / BLIP-2 (Q-Former)

Junnan Li, Dongxu Li, Silvio Savarese, Steven C. H. Hoi, Caiming Xiong

BLIP-2 keeps a pretrained image encoder and a pretrained LLM frozen and trains only a small Q-Former, a set of learnable query tokens that extract the visual features most relevant to text and feed them into the language model. This two-stage bridging cuts the trainable parameters and compute needed for vision-language pretraining by orders of magnitude, letting new image encoders and LLMs be combined into captioning and visual question-answering systems without full end-to-end retraining.

UnknownDifficulty 5/10Verified
Read record Primary source
0132023Multimodality

Technical report

Open Production VLMs (Qwen-VL/InternVL/PaLI/Kosmos)

Jinze Bai, Jingren Zhou, Zhe Chen, Jifeng Dai, Xi Chen, Radu Soricut, Shaohan Huang, Furu Wei

These models connect a strong vision encoder to a language model and train through staged image-text pretraining followed by multimodal instruction tuning, adding capabilities such as higher-resolution input, region grounding, OCR, and multilingual support. Represented by Qwen-VL, the family shipped weights, tooling, and reproducible recipes that made competitive image understanding, document reading, and grounded visual dialogue available to build on directly rather than only behind proprietary services.

UnknownDifficulty 5/10Verified
Read record Primary source
0142023Multimodality

Peer reviewed

SAM / Grounding DINO / ImageBind (perception foundation models)

Alexander Kirillov, Ross Girshick, Piotr Dollár, Shilong Liu, Lei Zhang, Rohit Girdhar, Ishan Misra

Segment Anything trains a promptable segmentation model on a billion-mask dataset so that a point, box, or text-linked prompt yields object masks for categories never explicitly labeled; Grounding DINO adds open-vocabulary detection from text queries, and ImageBind aligns six modalities into one embedding space. Represented by SAM, this family removed the need to collect labels and train a fresh model for each new segmentation or detection target, enabling zero-shot and interactive perception that downstream systems can prompt directly.

UnknownDifficulty 5/10Verified
Read record Primary source
0152022Multimodality

Preprint

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen)

Chitwan Saharia, William Chan, Saurabh Saxena, Mohammad Norouzi

Imagen pairs a frozen T5-XXL language-model text encoder with a cascade of diffusion models and finds that scaling the text encoder improves image-text alignment more than scaling the image model. It reached strong photorealism and prompt fidelity on standard evaluations. It clarified the role of language understanding in text-to-image systems.

UnknownDifficulty 6/10Verified
Read record Primary source
0162024Multimodality

Technical report

Video Generation Models as World Simulators (Sora)

Tim Brooks, Bill Peebles, OpenAI

Sora represents videos as spacetime patches and trains a diffusion Transformer over them, scaling data and compute rather than baking in physics. The technical report shows long, high-resolution, temporally consistent clips and argues that scale yields rudimentary simulation of objects and dynamics. It signaled diffusion-Transformer scaling as a path to video and world models.

UnknownDifficulty 5/10Verified
Read record Primary source