MMOOZZGG.com

Glossary

What is pretraining

In short

Pretraining is the first large stage in building a language model, in which it learns to predict the continuation of a text from a huge collection of books, websites and code.

Language models · pretraining

MMOOZZGG.com editorsChecked: 2 min read

For a child

Before someone becomes a doctor or a cook, they go to school for many years and read about everything: planets, animals, history. This is not yet a profession, but without this base they cannot learn one. A neural network also first reads mountains of different texts for a long time, to understand how language works. It learns a specific job only later.

In plain words

At first the model knows nothing. It is shown a huge amount of text from the internet, books and program code, with one task: guess the next piece of text. If it is wrong, it adjusts its internal settings a little; if it is right, it moves on. Nobody labels the examples by hand: the text itself is the teacher. Over billions of such attempts, the model learns grammar, facts about the world and the first signs of reasoning. After this stage it can continue a text but follows instructions poorly: it may answer a question with a new question. Later stages turn it into a polite assistant.

For business

This stage needs huge computing clusters and a long time, so companies almost never run it themselves. A business takes a ready model, open or cloud, and adapts it to its tasks with instructions, document search or fine-tuning. Pretraining has two consequences that matter. The model’s knowledge ends at the date the data was collected, so fresh prices and rules must go into the request. And the model has inherited the errors and biases of its training texts. Before you choose, check the license for the weights, the knowledge cutoff date and the quality in the languages you need. Teams usually compare ready models on their own tasks during a pilot.

For a specialist

Pretraining means training a model on a large unlabeled corpus with a self-supervised objective. For decoder models, the goal is next-token prediction (causal language modeling). BERT (Devlin and co-authors, 2018) learned to restore masked tokens (masked language modeling); then one output layer was added and the model was fine-tuned for a task. Brown and co-authors (2020) showed that GPT-3, after pretraining alone, can handle new tasks when you show it a few examples right in the text of the prompt. Hoffmann and co-authors (2022) argued that, for a fixed computing budget, the model size and the number of training tokens should grow in equal proportion. The key decisions at this stage are the makeup and filtering of the corpus, deduplication, the share of code and languages, the tokenizer and the learning rate schedule. The result is called a base model, and instruction tuning and alignment come after it.

Example

A model that has only been pretrained may answer the request “Write a letter to a client about a delayed delivery” by continuing a list of similar tasks instead of writing the letter. After instruction tuning, the same model writes the letter itself.

Related terms

Questions and answers

How is pretraining different from fine-tuning? Pretraining gives a model general knowledge of language and the world from a huge body of text. Fine-tuning then adapts the ready model to a specific style, format or task on a much smaller set of examples.
Can a company pretrain its own model? Technically yes, but a model at the level of modern chatbots needs huge text collections and computing clusters. It almost always pays more to take a ready open-weight model and fine-tune it, or connect it to a document search.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us