Chatbot Arena / LLM-as-Judge (MT-Bench/AlpacaEval)

Editorial record

Plain-language summary

Chatbot Arena collects anonymous head-to-head comparisons: users chat with two unnamed models, vote for the better response, and the votes feed an Elo-style rating that ranks models. MT-Bench is a set of multi-turn questions scored by a strong model acting as an automated judge, which the paper validates against human preferences while also documenting judge biases such as favoring longer answers or the first response shown. Together they gave the field a way to evaluate conversational quality and instruction following that static multiple-choice benchmarks cannot capture.