Native/Omni Multimodality (Gemini / GPT-4V / GPT-4o)

Editorial record

Plain-language summary

The report describes models built to accept and reason over mixed inputs (text, images, audio, and video) natively and to produce text and image outputs, trained jointly across modalities and offered in sizes from on-device to datacenter scale. Native multimodal training is meant to avoid the information loss of connecting independently trained encoders after the fact. The report reports benchmark results across language, coding, reasoning, and multimodal tasks; the underlying weights and full training details are not public, and related closed systems like GPT-4V and GPT-4o pursue similar goals.