LLaVA: Visual Instruction Tuning

Editorial record

Plain-language summary

Image features from a frozen CLIP encoder are mapped into the language model's token space by a single trainable projection, so the model can treat an image as extra context alongside a text prompt. The team used a text-only GPT model to generate multimodal instruction-following examples (questions, descriptions, and reasoning about images) and fine-tuned on them. This showed that a capable image-chat assistant could be built cheaply from existing open components plus synthetic instruction data, and it became a common baseline architecture for open multimodal models.