Glossary
What is knowledge distillation?
In short
Knowledge distillation is a way to train a small model to repeat the answers of a large one, to get almost the same result cheaper and faster.
Language models · distillation
MMOOZZGG.com editorsChecked: 2 min read
For a child
An older brother studied problem solving for a long time and knows many tricks. He has no time to retell the whole subject, so he shows his younger brother how he solves examples himself and hints which answers are almost right. The younger one learns faster than from a thick textbook. A small neural network learns from a large one in exactly the same way.
In plain words
Large neural networks are smart but slow and expensive. Distillation moves their knowledge into a small model. The large one is called the teacher, and the small one the student. The teacher answers thousands of questions, and the student practices answering the same way. What matters is that the student sees not only the final answer but also how sure the teacher is about each option. From this it learns more than from a bare list of correct answers. The result is a compact model that works faster and is almost as good as the teacher on the tasks you need.
For business
This technique lowers the cost of each answer. The large model is needed once, to label the examples, and then the small one works: it answers customers faster and fits on an inexpensive server. This is handy when the task is narrow and repeats thousands of times a day: sorting requests, extracting fields from documents, short support answers. The costs are generating training examples, the training itself and testing. Risks: the student inherits the teacher’s errors and answers noticeably worse outside its task. Read the teacher model’s terms of use separately: they may restrict training other models on its answers. The payback of such a pair is calculated from the cost per token on your own server.
For a specialist
Knowledge distillation trains a compact student model to reproduce the output distribution of a teacher model. Hinton, Vinyals and Dean (2015) proposed training the student on the teacher’s soft probabilities, smoothed by a temperature in the softmax: they hold information about class similarity that hard labels lack. The loss usually combines the Kullback-Leibler divergence with the teacher’s outputs and cross-entropy with the true labels. Variants: on logits, on intermediate representations, and at the sequence level on texts generated by the teacher. DistilBERT (Sanh et al., 2019) made BERT 40% smaller, kept about 97% of its language understanding quality and ran 60% faster. The authors of DeepSeek-R1 (2025) showed that the reasoning skills of a large model can be moved into small models this way. The method is combined with quantization and pruning.
Do not confuse. Not to be confused with distillation in chemistry, where the word means separating liquids by heating.
Example
An online store gets tens of thousands of emails a day. First a large model labels a few thousand requests by topic, then a small one is trained on these labels. It sorts the mail on the store’s server in fractions of a second.
Related terms
Questions and answers
How is distillation different from fine-tuning?
Can you distill a model for your own task?
Sources
- Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network, 2015arxiv.org
- Sanh, Debut, Chaumond, Wolf, DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019arxiv.org
- DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025arxiv.org
Not sure whether your company needs this?
AI solutions map