MMOOZZGG.com

Glossary

What is a foundation model

In short

A foundation model is a large neural network trained on a huge amount of general data, which teams then adapt to many specific tasks.

Language models · foundation model

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine a big construction set. From the same parts you can build a car, a castle or a rocket, if you add a few special blocks and a manual. You do not need to build from zero every time. A foundation model is like this set: people make helpers for very different jobs from it.

In plain words

In the past a separate neural network was trained for each task: one for translation, another for finding spam. A foundation model is first trained for a long time on a huge amount of text, images or code, and it gains general knowledge of language and the world. Then the right helper is made from it quickly: a chatbot, a translator, a document checking system. A little fine-tuning or a good instruction is enough for this. It is like a school graduate: the general education is already there, and a profession takes months to learn, not years.

For business

Companies almost never train such a base themselves: it costs huge sums and needs data centers. It pays more to take a ready model and adapt it: give it instructions, connect documents through search and, if needed, fine-tune it on your own examples. The choice of base decides a lot: quality in Russian, the length of documents it reads, the price of a request, and whether it can run on your own servers. The risk is dependence: if all solutions are built on one model, every service of the company inherits its errors and bias, and changing the vendor is expensive. So the architecture is built so that the base can be replaced. A separate analysis explains the layers of AI in a company, from the model to the processes.

For a specialist

The term “foundation model” was introduced by researchers at the Stanford CRFM center in the paper by Bommasani et al. (2021): a model trained on broad data at scale, usually by self-supervised learning, and adaptable to a wide range of downstream tasks. Examples in the paper are BERT, GPT-3 and DALL-E. The authors named two properties: new abilities appear as scale grows (emergence), and homogenization, where all systems adapted from one base inherit its defects. Adaptation goes through fine-tuning, instruction tuning, parameter-efficient methods (PEFT, LoRA), prompting and RAG. The word base model also means the version of a model right after pretraining, before instruction tuning and alignment, as opposed to the instruct or chat version.

Example

A company takes a ready foundation model and in two weeks makes an assistant for its purchasing department from it: it writes instructions and connects the supplier catalog and contracts. The model did not have to be trained from zero.

Related terms

Questions and answers

How is a foundation model different from an ordinary neural network? An ordinary neural network is trained for one task. A foundation model is trained on broad data once and then adapted to dozens of different tasks without training from zero.
How is a foundation model different from a chat model? A chat model is made from a foundation model: it is further trained to follow instructions and hold a dialog. Without this tuning the model simply continues the text it was given.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us