Training Compute-Optimal Large Language Models (Chinchilla)

Editorial record

Plain-language summary

DeepMind trained over 400 models at varied size and token counts and re-estimated the compute-optimal frontier, finding that for a given compute budget parameters and data should grow together at about a 1:1 ratio rather than favoring size. To demonstrate it they trained Chinchilla, a 70B model on 1.4 trillion tokens, which outperformed the 280B Gopher and other larger models while being cheaper to run at inference. The result redirected the field toward training smaller models on far more data.