MMOOZZGG.com

Glossary

What is instruction tuning

In short

Instruction tuning is the further training of a language model on pairs of “instruction and correct answer”. After it, the model carries out requests instead of just continuing the text.

Language models · instruction tuning

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine a student who has read the whole school library but answers the teacher’s question by retelling a random page. So the teacher shows many times: here is the task, and here is how to answer it correctly. Soon the student understands that an answer to the question is expected, not just any page. People do the same with a neural network.

In plain words

After pretraining, a model knows a lot but can only continue text. Ask it to “write three ideas for a promotion”, and it may add a dozen similar requests instead of ideas. To fix this, it is shown thousands of examples: a task in human language and a model answer. The tasks vary: translate, shorten, answer a question, make a list. The model learns a general rule: the text after an instruction must carry it out. The result is a version that developers mark with the word instruct or chat. This is the version that chatbots use.

For business

Almost every model that companies connect through an API or install on their own server has already gone through this tuning at the developer. Your own tuning makes sense when the model must give answers in your house format, know internal terms or work with narrow documents, and prompts are no longer enough. The main cost here is not GPUs but data: you need hundreds or thousands of checked pairs of “request and reference answer”, and the company’s experts prepare them. The model learns mistakes in the examples as carefully as correct answers. Start with a sample of real requests and test the current model on them: a good prompt and document search are often enough. Which data to collect in advance is decided when you prepare data for AI.

For a specialist

Instruction tuning is supervised fine-tuning (SFT) of a pretrained model on a set of tasks phrased as natural-language instructions. Wei and coauthors (2021) fine-tuned a 137-billion-parameter model on more than 60 tasks written as instruction templates. The resulting FLAN, in zero-shot mode, beat the 175-billion-parameter GPT-3 on 20 of 25 tested tasks. Chung and coauthors (2022) scaled the approach to 1.8 thousand tasks, added chain-of-thought data and published the Flan-T5 checkpoints. The loss function is the usual one, cross-entropy on the answer tokens, and the instruction tokens are usually masked. For chat models the data uses a dialog template with roles. In the InstructGPT scheme (Ouyang and coauthors, 2022) this stage comes before preference training, and today DPO is also used instead of RLHF at the preference stage.

Example

A law firm collects two thousand pairs of “client question and lawyer’s answer” and tunes an open model on them. After that, the model answers in the firm’s format: conclusion, reference to the article of the law, next step.

Related terms

Questions and answers

How is instruction tuning different from ordinary fine-tuning? It is one kind of fine-tuning. Ordinary fine-tuning can teach a model anything, such as a style or terms.

Here the goal is one: to teach the model to carry out requests written in human language.

Does a company need to do instruction tuning itself? Most often not: ready models have already gone through it at the developer. A company does its own when prompts and document search do not give the needed format or quality, and it has hundreds of checked examples.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us