MMOOZZGG.com

Glossary

What is direct preference optimization (DPO)?

In short

Direct preference optimization (DPO) is a method of fine-tuning a language model on pairs of answers, a good one and a bad one, that teaches it to answer the way people prefer, without a separate reward model.

Language models · direct preference optimization

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine someone shows you two drawings of the same house and says: I like this one more. Then another pair, and another. There are no scores, only a choice between two. After a hundred such pairs, you start drawing the way viewers like. A neural network is taught in a similar way: it is shown two of its answers, and someone marks which one is better.

In plain words

To make a model answer politely, to the point and without filler, it is shown pairs of answers to the same question. A person has marked which of the two is better. The method shifts the model directly: it raises the chances of answers like the chosen ones and lowers the chances of the rejected ones. At the same time the model is kept close to its original version, so it does not forget everything else. Before, this needed a separate judge model and a complex reinforcement learning setup. The new way works without a judge, so it is simpler and cheaper.

For business

The method made tuning a model to a company’s requirements possible for a small team. The raw material is simple: pairs of answers where an employee marked the better one. Such data often builds up by itself, for example when a support operator edits an AI draft before sending it: the draft becomes the rejected answer and the edited text the preferred one. Training runs on the same GPUs as regular fine-tuning, without a separate judge model. The risks are familiar: the model picks up the raters’ biases, and with weak control it starts to answer longer and more eagerly to please. Before training, agree on the rating criteria and measure quality on a held-out sample. The metric and the success threshold are fixed in the pilot.

For a specialist

Rafailov and colleagues proposed Direct Preference Optimization in 2023. In RLHF you first train a reward model on comparisons, then optimize the policy with reinforcement learning, with a penalty for moving away from the original model. The authors showed that the optimal policy of this task can be expressed through the reward in closed form, and they reduced training to a simple classification loss on pairs: minus the log of the sigmoid of β times the difference of the log probability ratios of the preferred and rejected answers between the trained model and the reference model. No sampling from the model is needed during training. In experiments the method beat PPO-based RLHF in sentiment control and matched it in summarization and single-turn dialog. In the TRL library an example is given by the fields prompt, chosen and rejected, β is 0.1 by default, and a larger value keeps the model closer to the reference. Other loss variants there include IPO, Robust DPO and the length normalization from the SimPO paper.

Example

An online store collects pairs: the AI draft of a reply and the version the operator sent to the customer after editing. After a month there are enough pairs to fine-tune the model with DPO, and operators edit the drafts less often.

Related terms

Questions and answers

How is DPO different from RLHF? In RLHF you first train a separate judge model and then reinforce the main model with its scores. DPO needs no judge: the main model learns directly from preference pairs, so training is simpler, more stable and needs less computing.
What data does DPO need? Pairs of answers to the same requests, with the preferred one marked. Raters collect them, or they come from staff edits, and the quality of the rating criteria matters more than the volume.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us