Flamingo: a Visual Language Model for Few-Shot Learning

Editorial record

Plain-language summary

Frozen vision and language backbones are bridged by a Perceiver-style resampler that turns variable numbers of image features into a fixed set of tokens, plus gated cross-attention layers inserted into the language model so text generation can attend to images. Training on large interleaved image-text web data teaches the bridge without retraining the expensive backbones. This allowed few-shot performance on captioning and visual question answering by showing examples in the prompt, rather than fine-tuning a separate model per task.