Evaluation & Benchmarks · 2021
Contamination-Resistant Benchmarks (TruthfulQA/LiveBench/SimpleQA)
A family of contamination-resistant benchmarks (TruthfulQA, LiveBench, SWE-bench-verified) designed so scores are not inflated by training-set memorization or leaked test data, using adversarial questions, continuously refreshed recent items, or human-verified solvable tasks.
Editorial record
Plain-language summary
Because static public benchmarks leak into training corpora, high scores can reflect memorization rather than capability. These efforts attack that differently: TruthfulQA targets questions where imitating human text produces confident falsehoods, LiveBench continuously draws fresh questions from recent sources (papers, news, competitions) with objective ground truth and rotates them out to limit contamination, and SWE-bench-verified is a human-filtered subset of real GitHub issues confirmed to be well-specified and solvable. The shared idea is to keep evaluation valid over time by making the test set hard to have already seen or trivially matched.
Source record
Provenance
- Record ID
- P-448
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2406.19314
- arXiv:2406.19314
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.