MMOOZZGG.com

Glossary

What is transfer learning

In short

Transfer learning is an approach in which a model already trained on one large task is adapted to a new task using a small dataset.

Machine learning · transfer learning

MMOOZZGG.com editorsChecked: 2 min read

For a child

If you can ride a scooter, it is easier to learn to ride a bike: you already keep your balance and watch the road. You do not have to start from zero; you only need to get used to the pedals. A computer learns in a similar way. First it spends a long time learning to look at pictures in general, and then it quickly learns what your own toys look like.

In plain words

Training a model from scratch is expensive: it needs huge numbers of examples and a lot of computing. So teams take a model that has already learned general things, such as telling apart lines, shapes and textures in photos or understanding how a language works. They change its last layers for their task and train it further on hundreds or thousands of their own examples. This is how a clinic gets a model for medical images and a factory gets one for photos of parts, even though each has little data of its own. Large language models follow the same principle: first they learn from a huge body of text, then they are adjusted.

For business

This approach explains why mid-size companies can afford AI at all: there is no need to collect millions of examples and rent computing for months. A company takes an open or commercial base model and trains it further on its own data; sometimes a few hundred labeled examples already give a working result. The costs come from preparing data, GPU time and quality checks. The risks: the original model may not know your industry, it carries over other people’s errors and bias, and its license may limit commercial use. Before choosing a model, check the license, where the data is stored during training and the quality on your examples. How such models fit into a company’s work is shown in the review of AI in a company floor by floor.

For a specialist

Transfer learning uses representations learned on a source task for a target task with less data. The classic scheme: a network is pretrained on a large corpus, then either frozen as a feature extractor (feature extraction) or trained further in full or in part (fine-tuning). Yosinski et al. (2014) showed that the early layers of convolutional networks learn general features while the later ones are task-specific, and that initializing with transferred weights improves generalization even after fine-tuning. In NLP, the ULMFiT method (Howard, Ruder, 2018) with one hundred labeled examples matched the quality of training from scratch on one hundred times more data. Today the scheme is pretraining a base model and then fine-tuning it, including with parameter-efficient methods (PEFT). Related concepts: domain adaptation, multi-task learning and few-shot learning.

Example

A chain of car repair shops wants to identify the type of body damage from a photo. Instead of training from scratch, engineers take a model already trained on a large set of ordinary photos and train it further on a few thousand images from their own work orders.

Related terms

Questions and answers

How is transfer learning different from fine-tuning? Transfer learning is the general idea: use what a model learned on another task. Fine-tuning is one way to do it, when the weights of a ready model are adjusted on your data.
How much data do you need if the model is already trained? Usually many times less than for training from scratch: for a simple classification, a few hundred examples are sometimes enough. An experiment shows the exact amount, so you start with a small sample and watch how quality grows.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us