Glossary
What is instruction tuning
In short
Instruction tuning is the further training of a language model on pairs of “instruction and correct answer”. After it, the model carries out requests instead of just continuing the text.
Language models · instruction tuning
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a student who has read the whole school library but answers the teacher’s question by retelling a random page. So the teacher shows many times: here is the task, and here is how to answer it correctly. Soon the student understands that an answer to the question is expected, not just any page. People do the same with a neural network.
In plain words
After pretraining, a model knows a lot but can only continue text. Ask it to “write three ideas for a promotion”, and it may add a dozen similar requests instead of ideas. To fix this, it is shown thousands of examples: a task in human language and a model answer. The tasks vary: translate, shorten, answer a question, make a list. The model learns a general rule: the text after an instruction must carry it out. The result is a version that developers mark with the word instruct or chat. This is the version that chatbots use.
For business
Almost every model that companies connect through an API or install on their own server has already gone through this tuning at the developer. Your own tuning makes sense when the model must give answers in your house format, know internal terms or work with narrow documents, and prompts are no longer enough. The main cost here is not GPUs but data: you need hundreds or thousands of checked pairs of “request and reference answer”, and the company’s experts prepare them. The model learns mistakes in the examples as carefully as correct answers. Start with a sample of real requests and test the current model on them: a good prompt and document search are often enough. Which data to collect in advance is decided when you prepare data for AI.
For a specialist
Instruction tuning is supervised fine-tuning (SFT) of a pretrained model on a set of tasks phrased as natural-language instructions. Wei and coauthors (2021) fine-tuned a 137-billion-parameter model on more than 60 tasks written as instruction templates. The resulting FLAN, in zero-shot mode, beat the 175-billion-parameter GPT-3 on 20 of 25 tested tasks. Chung and coauthors (2022) scaled the approach to 1.8 thousand tasks, added chain-of-thought data and published the Flan-T5 checkpoints. The loss function is the usual one, cross-entropy on the answer tokens, and the instruction tokens are usually masked. For chat models the data uses a dialog template with roles. In the InstructGPT scheme (Ouyang and coauthors, 2022) this stage comes before preference training, and today DPO is also used instead of RLHF at the preference stage.
Example
A law firm collects two thousand pairs of “client question and lawyer’s answer” and tunes an open model on them. After that, the model answers in the firm’s format: conclusion, reference to the article of the law, next step.
Related terms
Questions and answers
How is instruction tuning different from ordinary fine-tuning?
Sources
Not sure whether your company needs this?
AI solutions map