Glossary
RLHF: reinforcement learning from human feedback
In short
RLHF (reinforcement learning from human feedback) is a stage of training a language model in which people compare its answers and the model learns to give the answers they prefer.
Language models · reinforcement learning from human feedback
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a puppy learning commands. When it does the right thing, its owner praises it and gives a treat; when it gets it wrong, it gets nothing. Soon the puppy understands what is expected. A neural network is similar: people choose which of its answers is more polite and useful, and it slowly learns to answer the way they like.
In plain words
After the first training, a model can continue text, but it may answer rudely, dodge the question or make things up. To fix this, human raters look at several of its answers to one question and pick the better one. From many such comparisons, a separate judge model is built that predicts which answer a person will like. Then the main model is trained to get high scores from the judge. For example, when asked to help with a letter, it starts writing the letter itself instead of talking about letters in general.
For business
Thanks to this stage, chat models follow instructions, refuse harmful requests and keep a polite tone. Ouyang and co-authors (2022) showed the power of the method: people preferred the answers of the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3. A company can draw two conclusions. First, outside raters with their own rules shaped much of a ready model’s behavior, so it may refuse normal work tasks or please the user at the cost of accuracy. Second, your own preference stage makes sense when you have a flow of ratings from employees, such as “helpful” and “did not help” buttons in an assistant. Before that, check who gives the ratings and by what criteria: their biases will pass into the model.
For a specialist
Christiano and co-authors (2017) laid the basis of RLHF (reinforcement learning from human feedback) on Atari games and simulated robot motion. The agent learned the reward function from comparisons of pairs of trajectory segments and got feedback on less than 1% of its interactions with the environment. For language models, Ouyang and co-authors (2022) set the scheme in InstructGPT: supervised fine-tuning on demonstrations (SFT), human ranking of answers, training a reward model, and policy optimization with PPO with a KL-divergence penalty for drifting from the original model. The method’s problems are reward hacking, which exploits flaws of the reward model, sycophancy and costly labeling. Alternatives: DPO (Rafailov and co-authors, 2023) turns the task into a classification loss on preference pairs, with no separate reward model and no RL, and RLAIF replaces human ratings with ratings from another model.
Example
A legal assistant first answered the request “draft a claim letter” with general advice. After a training stage on ratings, where lawyers picked the more useful of two answers, it started sending a ready claim letter right away.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map