CodeT5 / CodeGen / InCoder / Fill-in-the-Middle

Editorial record

Plain-language summary

CodeT5 (encoder-decoder, identifier-aware), CodeGen (autoregressive, multi-turn program synthesis), and InCoder (masked-span infilling) explored how objective and architecture shape code models. The FIM paper showed a simple data transformation: move a randomly chosen middle span to the end with sentinel tokens, so a standard causal model learns to fill gaps using surrounding code while retaining normal left-to-right ability. This removed the need for a separate bidirectional model to power editor-style completions like inserting a function body between existing lines.