The Evaluation Crisis (contamination/saturation/judge-bias)

Editorial record

Plain-language summary

The work documents three linked failures in how models are scored: contamination, where test items leak into training data so results overstate ability; saturation, where top models cluster near the ceiling of older benchmarks so the numbers no longer separate them; and judge bias, where using a model as an automated grader introduces systematic errors like preferring verbose or self-similar answers. By characterizing these effects, it argues that many headline evaluation numbers are unreliable and motivates contamination checks, harder benchmarks, and more careful judging protocols.