Preference Optimization Family (IPO/KTO/ORPO/SimPO)

Editorial record

Plain-language summary

Each variant targets a specific limitation of DPO: IPO replaces the log-sigmoid objective with a bounded one to curb overfitting when preferences are near-deterministic; KTO learns from single labeled examples marked desirable or undesirable rather than requiring paired comparisons; ORPO folds a preference odds-ratio penalty directly into supervised fine-tuning so no separate alignment stage or reference model is needed; SimPO removes the reference model and uses a length-normalized reward with a target margin. Together they made preference-based alignment cheaper and more stable, reducing reliance on paired data, reference models, and reward-model training.