Glossary
What is transfer learning
In short
Transfer learning is an approach in which a model already trained on one large task is adapted to a new task using a small dataset.
Machine learning · transfer learning
MMOOZZGG.com editorsChecked: 2 min read
For a child
If you can ride a scooter, it is easier to learn to ride a bike: you already keep your balance and watch the road. You do not have to start from zero; you only need to get used to the pedals. A computer learns in a similar way. First it spends a long time learning to look at pictures in general, and then it quickly learns what your own toys look like.
In plain words
Training a model from scratch is expensive: it needs huge numbers of examples and a lot of computing. So teams take a model that has already learned general things, such as telling apart lines, shapes and textures in photos or understanding how a language works. They change its last layers for their task and train it further on hundreds or thousands of their own examples. This is how a clinic gets a model for medical images and a factory gets one for photos of parts, even though each has little data of its own. Large language models follow the same principle: first they learn from a huge body of text, then they are adjusted.
For business
This approach explains why mid-size companies can afford AI at all: there is no need to collect millions of examples and rent computing for months. A company takes an open or commercial base model and trains it further on its own data; sometimes a few hundred labeled examples already give a working result. The costs come from preparing data, GPU time and quality checks. The risks: the original model may not know your industry, it carries over other people’s errors and bias, and its license may limit commercial use. Before choosing a model, check the license, where the data is stored during training and the quality on your examples. How such models fit into a company’s work is shown in the review of AI in a company floor by floor.
For a specialist
Transfer learning uses representations learned on a source task for a target task with less data. The classic scheme: a network is pretrained on a large corpus, then either frozen as a feature extractor (feature extraction) or trained further in full or in part (fine-tuning). Yosinski et al. (2014) showed that the early layers of convolutional networks learn general features while the later ones are task-specific, and that initializing with transferred weights improves generalization even after fine-tuning. In NLP, the ULMFiT method (Howard, Ruder, 2018) with one hundred labeled examples matched the quality of training from scratch on one hundred times more data. Today the scheme is pretraining a base model and then fine-tuning it, including with parameter-efficient methods (PEFT). Related concepts: domain adaptation, multi-task learning and few-shot learning.
Example
A chain of car repair shops wants to identify the type of body damage from a photo. Instead of training from scratch, engineers take a model already trained on a large set of ordinary photos and train it further on a few thousand images from their own work orders.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map