Direct Preference Optimization (DPO)

Editorial record

Plain-language summary

Direct Preference Optimization uses the mathematical link between reward and optimal policy to rewrite preference learning so the language model itself is optimized directly on chosen-versus-rejected examples with a simple supervised loss. There is no reward model to train and no PPO sampling, so it is more stable and cheaper while matching or beating PPO-based RLHF on preference tuning. This made preference alignment practical for teams without reinforcement-learning infrastructure.