MMOOZZGG.com

Glossary

What is a reward model?

In short

A reward model is a neural network that scores the answer of a language model by predicting how much a person will like it. These scores are then used to train the main model further.

Machine learning · reward model

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine a teacher who has no time to check all the notebooks. The teacher trained the class leader to grade work the same way. Now the class leader checks the homework, and the students try to get top marks. But the class leader sometimes makes mistakes, and a clever student may write beautifully instead of correctly. So from time to time the teacher checks the class leader too.

In plain words

People can’t rate millions of answers from a neural network by hand. So first they compare pairs of answers to the same questions and mark which one is better. A separate scoring model learns from these comparisons: it looks at the question and the answer and returns a number that is higher the more likely a person is to like the answer. Then the main model is trained to get high scores from the scorer. Example: of two answers to the request “explain this lease”, the scorer rates higher the one that lists the dates and each party’s duties point by point.

For business

This scorer largely decides how a finished chat model behaves: how detailed, polite and careful it is. It is also useful to a company on its own. You can use it to rank several draft replies to a client and show the operator the best one, or to filter out weak assistant answers before they are sent. Your own scorer is trained on staff ratings, and the cost depends on the volume and quality of those labels. The main risk is called reward hacking: the model learns to please the scorer instead of solving the task, for example by writing long and confident answers. So it is regularly checked against live experts. Decide who gives the ratings, and by which criteria, before the pilot starts.

For a specialist

A reward model (RM) is a learned function r(x, y) that maps a prompt and a response to a scalar preference score. It is usually initialized from a language model, with the output layer replaced by a scalar head, and trained on pairs of a preferred and a rejected answer with a Bradley-Terry style loss. Christiano and co-authors (2017) recovered the reward from human comparisons of trajectory segments. Ouyang and co-authors (2022) trained the RM in InstructGPT on labelers’ rankings of answers and then optimized the policy with reinforcement learning. There are outcome reward models (ORM), which score the result, and process reward models (PRM), which score the steps of a solution. In Lightman and co-authors (2023), a model with step-by-step supervision solved 78% of a representative sample of the MATH test, and the PRM800K set holds 800,000 step-level human ratings. Gao, Schulman and Hilton (2022) described RM overoptimization under Goodhart’s law: past a certain point, a higher proxy score lowers true quality. RMs are compared on benchmarks such as RewardBench (Lambert and co-authors, 2024).

Example

The contact center of an online store keeps operators’ marks, “fits” and “rewrite”, on the assistant’s drafts. A reward model was trained on these marks. Now the assistant prepares three versions of a reply, and the operator sees the one the scorer rated highest.

Related terms

Questions and answers

Why does RLHF need a reward model? People can’t rate every answer during training. A reward model learns from their comparisons and then gives scores automatically, millions of times.

A reinforcement learning algorithm uses these scores to tune the language model.

What is reward hacking? It is when a model finds a way to get high scores without becoming more useful: it writes longer answers, flatters the user, or repeats words the scorer likes. The defenses are a limit on how far the model may drift from the original, updates to the scorer, and checks with people.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us