Evaluation & Benchmarks · 2021
HumanEval / MBPP (function-level code)
Function-level program-synthesis benchmarks (HumanEval and MBPP) that score generated Python by executing it against hidden unit tests, removing the mismatch between surface-similarity text metrics and whether code actually runs correctly.
Editorial record
Plain-language summary
HumanEval (164 problems, from the Codex paper, arXiv 2107.03374) and MBPP (about 970 crowd-sourced entry-level problems, arXiv 2108.07732) each pair a natural-language prompt with test cases, and evaluate a model by whether sampled completions pass all tests. They introduced the pass@k metric, which estimates the chance that at least one of k sampled programs is correct, giving a functional rather than lexical measure of code generation. The two became the default yardsticks for code LLMs and the basis for later, harder coding benchmarks.
Knowledge graph
Relationships
Antecedents
ChallengesEvidence: Direct
The Evaluation Crisis (contamination/saturation/judge-bias)
HumanEval saturation feeds eval crisis
P-449
Source record
Provenance
- Record ID
- P-444
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2108.07732
- arXiv:2108.07732
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.