Glossary
What is AI alignment?
In short
AI alignment is the tuning of an AI system so that it behaves as people intend: helpful, honest and harmless.
Language models · alignment
MMOOZZGG.com editorsChecked: 1 min read
For a child
A puppy can run and bark loudly, but at first it does not know the house rules. People teach it patiently: they praise it when it brings the ball and do not praise it when it chews slippers. Step by step it learns what is good and what is not. A neural network also learns to behave well, and this is called alignment.
In plain words
After training on texts from the internet, a neural network can continue any phrase, but it does not understand what people want from it. It may be rude, make up a fact or help with something dangerous. Alignment teaches it to follow people’s intentions. People compare pairs of answers and mark which one is more helpful and honest, and the model adjusts to these ratings. It also gets a set of rules to check itself against. It is like an intern: he knows a lot, but a mentor explains the rules for talking to customers.
For business
This tuning decides whether you can let the model talk to customers. A well-aligned model refuses dangerous requests, is not rude and admits when it does not know the answer instead of making one up. For a company, this is a matter of reputation and legal risk. But the developer’s general tuning does not know your rules: what you must never promise a customer, which topics to avoid, how to talk about prices. You add them on top with a system prompt, input and output filters and answer checks. Before launch, test the model on purpose with hard and provocative questions. It helps to sort AI risks for a business by type: data, errors, vendors, law.
For a specialist
Alignment covers the tasks and methods that bring a model’s behavior in line with human intentions and values: helpfulness, honesty, harmlessness. The basic pipeline is described in the InstructGPT paper (Ouyang et al., 2022): supervised fine-tuning (SFT) on demonstrations, then a reward model trained on human rankings of answers, then policy optimization with reinforcement learning (RLHF). Raters preferred the answers of the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3. Constitutional AI (Bai et al., 2022) replaces human labels of harm with a list of principles and feedback from AI (RLAIF). DPO (Rafailov et al., 2023) rewrites the RLHF task so that the model is optimized on preference pairs with a simple classification loss, without a separate RL loop. Open problems: reward hacking, sycophancy and jailbreaks that get around the limits.
Do not confuse. Not to be confused with text alignment in documents and on websites.
Example
A company tests a chatbot before launch. A user asks it to insult a competitor, and the aligned model politely refuses. When it does not know the delivery date, it says so instead of making up a date.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map