Evaluation & Benchmarks · 2024
SWE-bench (repository-level SWE)
Evaluates language models on resolving real GitHub issues by requiring a repository-wide code patch that makes the project's actual test suite pass, removing the artificiality of self-contained single-function coding tasks.
Editorial record
Plain-language summary
The benchmark draws over 2,000 issue-and-pull-request pairs from popular Python repositories, giving the model an issue description and the codebase and asking it to produce a patch, then judging it by running the maintainers' fail-to-pass and pass-to-pass tests. Success demands navigating a large codebase, editing multiple files, and understanding project context, and at release even strong models solved only a low single-digit percentage. It became a leading measure of practical software-engineering ability and drove work on agentic coding systems.
Source record
Provenance
- Record ID
- P-445
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2310.06770
- arXiv:2310.06770
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.