Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

5 records

0012021Code Models

Technical report

Codex: Evaluating LLMs Trained on Code (HumanEval)

Mark Chen, Jerry Tworek, Heewoo Jun, et al.

The model is a GPT language model further trained on a large corpus of GitHub code so it can turn a function signature and docstring into a working implementation. Rather than scoring generated code by similarity to a reference, HumanEval defines 164 hand-written problems with unit tests and reports pass@k, the chance that at least one of k sampled solutions passes all tests. This functional metric, plus the finding that sampling many candidates raises success rates, shaped how code-generation models are evaluated and underpinned tools like GitHub Copilot.

UnknownDifficulty 5/10Verified
Read record Primary source
0022022Code Models

Peer reviewed

AlphaCode (Competition-Level Code Generation)

Yujia Li, David Choi, Junyoung Chung, et al.

AlphaCode generates a very large pool of candidate programs per problem (up to millions), then filters them by running the provided example tests and clusters the survivors by their behavior on generated inputs to pick a small number to submit. This sample-filter-cluster loop, rather than a single accurate decode, is what let it reach an average ranking around the top 54% of participants on Codeforces contests. It demonstrated that hard, multi-step reasoning problems could be approached by trading inference-time compute for correctness when an execution oracle is available.

UnknownDifficulty 6/10Verified
Read record Primary source
0032022Code Models

Peer reviewed

CodeT5 / CodeGen / InCoder / Fill-in-the-Middle

Yue Wang, Steven C.H. Hoi, Erik Nijkamp, Caiming Xiong, Daniel Fried, Mike Lewis, Mohammad Bavarian, Mark Chen

CodeT5 (encoder-decoder, identifier-aware), CodeGen (autoregressive, multi-turn program synthesis), and InCoder (masked-span infilling) explored how objective and architecture shape code models. The FIM paper showed a simple data transformation: move a randomly chosen middle span to the end with sentinel tokens, so a standard causal model learns to fill gaps using surrounding code while retaining normal left-to-right ability. This removed the need for a separate bidirectional model to power editor-style completions like inserting a function body between existing lines.

UnknownDifficulty 5/10Verified
Read record Primary source
0042023Code Models

Technical report

Open Code Models (StarCoder/Code Llama/DeepSeek-Coder/Qwen-Coder)

Raymond Li, Harm de Vries, Leandro von Werra, Baptiste Rozière, Gabriel Synnaeve, Daya Guo, Binyuan Hui

StarCoder was trained on The Stack, a permissively licensed GitHub corpus with PII redaction and an opt-out mechanism, and supports fill-in-the-middle plus an 8K-token window for repository-scale context. Code Llama (continued-pretraining of Llama 2 on code, long-context and instruct variants) and DeepSeek-Coder (repo-level pretraining with strong data curation) extended the same open, code-specialized line. Together they gave practitioners high-quality open-weight coding models with transparent data provenance, enabling local deployment, fine-tuning, and reproducible evaluation.

UnknownDifficulty 5/10Verified
Read record Primary source
0052022Code Models

Peer reviewed

Execution Feedback / Unit-Test Gen / Program Repair

Bei Chen, Jian-Guang Lou, Weizhu Chen, Xinyun Chen, Denny Zhou

Rather than treating a generated program as final, these methods execute it against unit tests (often model-generated) and feed the resulting pass/fail signals and error traces back to guide selection or iterative repair. Approaches include ranking candidates by test agreement, self-debugging from stack traces, and generating tests to expose faults. This grounds code models in an objective oracle, letting them fix functional bugs that pure next-token likelihood cannot detect.

UnknownDifficulty 6/10Verified
Read record Primary source