Glossary
What is a foundation model
In short
A foundation model is a large neural network trained on a huge amount of general data, which teams then adapt to many specific tasks.
Language models · foundation model
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a big construction set. From the same parts you can build a car, a castle or a rocket, if you add a few special blocks and a manual. You do not need to build from zero every time. A foundation model is like this set: people make helpers for very different jobs from it.
In plain words
In the past a separate neural network was trained for each task: one for translation, another for finding spam. A foundation model is first trained for a long time on a huge amount of text, images or code, and it gains general knowledge of language and the world. Then the right helper is made from it quickly: a chatbot, a translator, a document checking system. A little fine-tuning or a good instruction is enough for this. It is like a school graduate: the general education is already there, and a profession takes months to learn, not years.
For business
Companies almost never train such a base themselves: it costs huge sums and needs data centers. It pays more to take a ready model and adapt it: give it instructions, connect documents through search and, if needed, fine-tune it on your own examples. The choice of base decides a lot: quality in Russian, the length of documents it reads, the price of a request, and whether it can run on your own servers. The risk is dependence: if all solutions are built on one model, every service of the company inherits its errors and bias, and changing the vendor is expensive. So the architecture is built so that the base can be replaced. A separate analysis explains the layers of AI in a company, from the model to the processes.
For a specialist
The term “foundation model” was introduced by researchers at the Stanford CRFM center in the paper by Bommasani et al. (2021): a model trained on broad data at scale, usually by self-supervised learning, and adaptable to a wide range of downstream tasks. Examples in the paper are BERT, GPT-3 and DALL-E. The authors named two properties: new abilities appear as scale grows (emergence), and homogenization, where all systems adapted from one base inherit its defects. Adaptation goes through fine-tuning, instruction tuning, parameter-efficient methods (PEFT, LoRA), prompting and RAG. The word base model also means the version of a model right after pretraining, before instruction tuning and alignment, as opposed to the instruct or chat version.
Example
A company takes a ready foundation model and in two weeks makes an assistant for its purchasing department from it: it writes instructions and connects the supplier catalog and contracts. The model did not have to be trained from zero.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map