GPQA: Graduate-Level Google-Proof QA

Editorial record

Plain-language summary

Existing knowledge benchmarks had largely saturated and were answerable by search, so they no longer separated genuine reasoning from retrieval. GPQA collects questions in biology, physics, and chemistry authored by PhD-level experts, then filters to items where other experts agree on the answer but skilled non-experts fail even after extended Googling (validated 'Google-proof'). This yields a small, high-difficulty set that measures whether a model can reason at graduate level and serves as a hard target and a testbed for scalable-oversight research.