Learning to Predict by the Methods of Temporal Differences

Editorial record

Plain-language summary

Sutton showed how to learn multi-step predictions incrementally: instead of comparing a prediction only to the eventual outcome, TD updates each prediction using the difference between successive predictions along the way. This let learning happen online, step by step, and made better use of experience than earlier supervised methods. TD became the computational core of most later reinforcement learning, including value estimation in games and control.