Glossary
What is direct preference optimization (DPO)?
In short
Direct preference optimization (DPO) is a method of fine-tuning a language model on pairs of answers, a good one and a bad one, that teaches it to answer the way people prefer, without a separate reward model.
Language models · direct preference optimization
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine someone shows you two drawings of the same house and says: I like this one more. Then another pair, and another. There are no scores, only a choice between two. After a hundred such pairs, you start drawing the way viewers like. A neural network is taught in a similar way: it is shown two of its answers, and someone marks which one is better.
In plain words
To make a model answer politely, to the point and without filler, it is shown pairs of answers to the same question. A person has marked which of the two is better. The method shifts the model directly: it raises the chances of answers like the chosen ones and lowers the chances of the rejected ones. At the same time the model is kept close to its original version, so it does not forget everything else. Before, this needed a separate judge model and a complex reinforcement learning setup. The new way works without a judge, so it is simpler and cheaper.
For business
The method made tuning a model to a company’s requirements possible for a small team. The raw material is simple: pairs of answers where an employee marked the better one. Such data often builds up by itself, for example when a support operator edits an AI draft before sending it: the draft becomes the rejected answer and the edited text the preferred one. Training runs on the same GPUs as regular fine-tuning, without a separate judge model. The risks are familiar: the model picks up the raters’ biases, and with weak control it starts to answer longer and more eagerly to please. Before training, agree on the rating criteria and measure quality on a held-out sample. The metric and the success threshold are fixed in the pilot.
For a specialist
Rafailov and colleagues proposed Direct Preference Optimization in 2023. In RLHF you first train a reward model on comparisons, then optimize the policy with reinforcement learning, with a penalty for moving away from the original model. The authors showed that the optimal policy of this task can be expressed through the reward in closed form, and they reduced training to a simple classification loss on pairs: minus the log of the sigmoid of β times the difference of the log probability ratios of the preferred and rejected answers between the trained model and the reference model. No sampling from the model is needed during training. In experiments the method beat PPO-based RLHF in sentiment control and matched it in summarization and single-turn dialog. In the TRL library an example is given by the fields prompt, chosen and rejected, β is 0.1 by default, and a larger value keeps the model closer to the reference. Other loss variants there include IPO, Robust DPO and the length normalization from the SimPO paper.
Example
An online store collects pairs: the AI draft of a reply and the version the operator sent to the customer after editing. After a month there are enough pairs to fine-tune the model with DPO, and operators edit the drafts less often.
Related terms
Questions and answers
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map