Evaluation & Benchmarks · 2022
BIG-bench / BBH / HELM (holistic eval)
A family of broad-coverage evaluation efforts (BIG-bench, BIG-bench Hard, and HELM) that shifted LLM assessment from single-task accuracy to standardized measurement across many tasks and multiple metrics, exposing where scaling helps, where it doesn't, and trade-offs beyond accuracy.
Editorial record
Plain-language summary
As models grew, single-benchmark scores gave a narrow and often misleading picture of capability. BIG-bench assembled hundreds of diverse community tasks (with BBH isolating the subset where models then lagged humans), and HELM evaluated many models on a common set of scenarios reporting not just accuracy but calibration, robustness, fairness, bias, toxicity, and efficiency side by side. Together they made evaluation multi-dimensional and comparable across models, revealing capability gaps and cost/quality trade-offs that a lone accuracy number hides.
Source record
Provenance
- Record ID
- P-443
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2211.09110
- arXiv:2211.09110
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.