Glossary
What is supervised fine-tuning (SFT)?
In short
Supervised fine-tuning (SFT) is a stage at which a language model is trained further on pairs of a request and a model answer, so that it follows instructions in the needed format.
Language models · supervised fine-tuning
MMOOZZGG.com editorsChecked: 1 min read
For a child
You have read many books and know lots of words, but you cannot yet write a proper letter to your grandmother. The teacher shows you a few good letters: this is how you say hello, this is how you tell your news, this is how you say goodbye. You copy the examples and soon write on your own. A neural network is taught with examples like these too.
In plain words
After the first training on texts from the internet, a model can continue phrases but understands poorly what you want from it. So it is shown many examples: here is a question, here is the right answer written by a person. The model adjusts and starts to answer in the same way: to the point, in the right format, politely. This turns a “text continuer” into an assistant that follows requests. This step adds almost no new knowledge: it teaches the manner and the form of the answer.
For business
Companies use this method when instructions in the prompt are no longer enough: they need a stable answer format, the brand’s tone, or documents handled by internal rules. The main cost is not graphics cards but data. You need hundreds or thousands of good examples that your experts write and check, and the model will faithfully repeat bad ones. Other costs are training, storing your own version of the model and repeating the process when the base model changes. The risks: the model can lose some general skills, and personal data in the examples must be anonymized. New facts are not loaded this way; they need search over documents. Always compare the result with a well-written prompt on the same questions.
For a specialist
SFT (supervised fine-tuning) is further training of a pretrained model with a teacher on labeled pairs of an input and a target answer, with cross-entropy over the answer tokens; the request tokens are usually masked. The variant on varied instructions is called instruction tuning. In InstructGPT (Ouyang and co-authors, 2022), SFT on labelers’ demonstrations was the first step before the reward model and RLHF; evaluators preferred the answers of the 1.3-billion-parameter version to those of GPT-3 with 175 billion. In LIMA (Zhou and co-authors, 2023), a 65-billion-parameter model was fine-tuned on only 1,000 carefully chosen examples with no reinforcement learning; the authors argue that knowledge comes from pretraining and this stage teaches format. SFT is done by a full update of the weights or by PEFT methods such as LoRA and QLoRA. Typical problems are catastrophic forgetting, overfitting on a small set and test data leaking into the training data.
Example
An insurance company selected the answers of its experienced operators to frequent questions and fine-tuned an open model on them. The bot began to answer in the company’s own tone and in one pattern: the point, the deadline, the next step.
Related terms
Questions and answers
How is SFT different from RLHF?
How many examples does SFT need?
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map