Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

11 records

0012024Evaluation & Benchmarks

Benchmark paper

SWE-bench (repository-level SWE)

Carlos E. Jimenez, John Yang, et al.

The benchmark draws over 2,000 issue-and-pull-request pairs from popular Python repositories, giving the model an issue description and the codebase and asking it to produce a patch, then judging it by running the maintainers' fail-to-pass and pass-to-pass tests. Success demands navigating a large codebase, editing multiple files, and understanding project context, and at release even strong models solved only a low single-digit percentage. It became a leading measure of practical software-engineering ability and drove work on agentic coding systems.

UnknownDifficulty 5/10Verified
Read record Primary source
0022021Evaluation & Benchmarks

Benchmark paper

MMLU: Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, et al.

The benchmark collects roughly 16,000 exam-style questions at difficulties from high school to professional level and scores models without task-specific training. At release most models scored near the 25% random-chance baseline while the largest GPT-3 reached about 44%, with especially weak and poorly calibrated performance on subjects like law and morality. It became a standard headline number for comparing the general knowledge of frontier models.

UnknownDifficulty 4/10Verified
Read record Primary source
0032021Evaluation & Benchmarks

Benchmark paper

GSM8K / MATH (mathematical reasoning)

Karl Cobbe, Vineet Kosaraju, John Schulman, Dan Hendrycks, Jacob Steinhardt

MATH contains 12,500 competition-level problems, each with a worked solution, spanning subjects like algebra, geometry, and number theory at varying difficulty. Because answers require multi-step derivations rather than lookup, early large models scored very low, making it a clear measure of reasoning progress. It is often paired with GSM8K, a grade-school word-problem set, and together they became standard yardsticks for tracking improvements in chain-of-thought and mathematical reasoning.

UnknownDifficulty 4/10Verified
Read record Primary source
0042021Evaluation & Benchmarks

Benchmark paper

HumanEval / MBPP (function-level code)

Mark Chen, Wojciech Zaremba, Jacob Austin, Augustus Odena, Charles Sutton

HumanEval (164 problems, from the Codex paper, arXiv 2107.03374) and MBPP (about 970 crowd-sourced entry-level problems, arXiv 2108.07732) each pair a natural-language prompt with test cases, and evaluate a model by whether sampled completions pass all tests. They introduced the pass@k metric, which estimates the chance that at least one of k sampled programs is correct, giving a functional rather than lexical measure of code generation. The two became the default yardsticks for code LLMs and the basis for later, harder coding benchmarks.

UnknownDifficulty 4/10Verified
Read record Primary source
0052023Evaluation & Benchmarks

Benchmark paper

Chatbot Arena / LLM-as-Judge (MT-Bench/AlpacaEval)

Lianmin Zheng, Wei-Lin Chiang, Ion Stoica, Xuechen Li, Yann Dubois, Tatsunori B. Hashimoto

Chatbot Arena collects anonymous head-to-head comparisons: users chat with two unnamed models, vote for the better response, and the votes feed an Elo-style rating that ranks models. MT-Bench is a set of multi-turn questions scored by a strong model acting as an automated judge, which the paper validates against human preferences while also documenting judge biases such as favoring longer answers or the first response shown. Together they gave the field a way to evaluate conversational quality and instruction following that static multiple-choice benchmarks cannot capture.

UnknownDifficulty 4/10Verified
Read record Primary source
0062023Evaluation & Benchmarks

Peer reviewed

The Evaluation Crisis (contamination/saturation/judge-bias)

Oscar Sainz, Zhang, et al.

The work documents three linked failures in how models are scored: contamination, where test items leak into training data so results overstate ability; saturation, where top models cluster near the ceiling of older benchmarks so the numbers no longer separate them; and judge bias, where using a model as an automated grader introduces systematic errors like preferring verbose or self-similar answers. By characterizing these effects, it argues that many headline evaluation numbers are unreliable and motivates contamination checks, harder benchmarks, and more careful judging protocols.

UnknownDifficulty 5/10Verified
Read record Primary source
0072021Evaluation & Benchmarks

Critique

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, Margaret Mitchell

The paper contends that scaling training data and compute concentrates benefits, raises environmental and labor costs, encodes and amplifies bias from unaudited web text, and produces systems that stitch together form without meaning (the “stochastic parrot”). It calls for documentation, curation, and restraint. It became the defining critical counterpoint to the scale-first paradigm and remains contested.

UnknownDifficulty 4/10Verified
Read record Primary source
0082023Evaluation & Benchmarks

Benchmark paper

GPQA: Graduate-Level Google-Proof QA

David Rein, et al.

Existing knowledge benchmarks had largely saturated and were answerable by search, so they no longer separated genuine reasoning from retrieval. GPQA collects questions in biology, physics, and chemistry authored by PhD-level experts, then filters to items where other experts agree on the answer but skilled non-experts fail even after extended Googling (validated 'Google-proof'). This yields a small, high-difficulty set that measures whether a model can reason at graduate level and serves as a hard target and a testbed for scalable-oversight research.

UnknownDifficulty 4/10Verified
Read record Primary source
0092022Evaluation & Benchmarks

Benchmark paper

BIG-bench / BBH / HELM (holistic eval)

Aarohi Srivastava, Mirac Suzgun, Jason Wei, Percy Liang, Rishi Bommasani, Tony Lee

As models grew, single-benchmark scores gave a narrow and often misleading picture of capability. BIG-bench assembled hundreds of diverse community tasks (with BBH isolating the subset where models then lagged humans), and HELM evaluated many models on a common set of scenarios reporting not just accuracy but calibration, robustness, fairness, bias, toxicity, and efficiency side by side. Together they made evaluation multi-dimensional and comparable across models, revealing capability gaps and cost/quality trade-offs that a lone accuracy number hides.

UnknownDifficulty 5/10Verified
Read record Primary source
0102023Evaluation & Benchmarks

Benchmark paper

Agent/Web Benchmarks (AgentBench/WebArena/GAIA/tau-bench)

Xiao Liu, Jie Tang, Shuyan Zhou, Graham Neubig, Grégoire Mialon, Thomas Scialom, Shunyu Yao, Karthik Narasimhan

Static QA benchmarks cannot measure whether a model can plan and act over many steps against a stateful environment. WebArena provides self-hosted realistic websites where agents must navigate and complete tasks judged by end-state, AgentBench spans multiple environments (OS, database, web, games), GAIA poses real-world questions needing tool use and multi-hop reasoning, and tau-bench simulates tool-and-user customer-service dialogues. By scoring task completion in executable settings, they exposed a large gap between chat competence and reliable agentic execution.

UnknownDifficulty 5/10Verified
Read record Primary source
0112021Evaluation & Benchmarks

Benchmark paper

Contamination-Resistant Benchmarks (TruthfulQA/LiveBench/SimpleQA)

Stephanie Lin, Owain Evans, Colin White, Tom Goldstein, Jason Wei, William Fedus

Because static public benchmarks leak into training corpora, high scores can reflect memorization rather than capability. These efforts attack that differently: TruthfulQA targets questions where imitating human text produces confident falsehoods, LiveBench continuously draws fresh questions from recent sources (papers, news, competitions) with objective ground truth and rotates them out to limit contamination, and SWE-bench-verified is a human-filtered subset of real GitHub issues confirmed to be well-specified and solvable. The shared idea is to keep evaluation valid over time by making the test set hard to have already seen or trivially matched.

UnknownDifficulty 4/10Verified
Read record Primary source