Code Models · 2021
Codex: Evaluating LLMs Trained on Code (HumanEval)
Introduced Codex and the HumanEval benchmark, evaluating a GPT model fine-tuned on public code for generating programs from docstrings and measuring correctness by actually running unit tests, establishing execution-based functional evaluation instead of text-overlap metrics.
Editorial record
Plain-language summary
The model is a GPT language model further trained on a large corpus of GitHub code so it can turn a function signature and docstring into a working implementation. Rather than scoring generated code by similarity to a reference, HumanEval defines 164 hand-written problems with unit tests and reports pass@k, the chance that at least one of k sampled solutions passes all tests. This functional metric, plus the finding that sampling many candidates raises success rates, shaped how code-generation models are evaluated and underpinned tools like GitHub Copilot.
Knowledge graph
Relationships
Antecedents
ExtendsEvidence: Strongly supported
AlphaCode (Competition-Level Code Generation)
AlphaCode extends code gen with search
P-341
ExtendsEvidence: Direct
Open Code Models (StarCoder/Code Llama/DeepSeek-Coder/Qwen-Coder)
Open code models follow Codex
P-343
Descendants
Applies toEvidence: Direct
Improving Language Understanding by Generative Pre-Training (GPT)
Codex fine-tunes GPT on code
P-340
Source record
Provenance
- Record ID
- P-340
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2107.03374
- arXiv:2107.03374
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.