MMOOZZGG.com

Glossary

What is a transformer model?

In short

A transformer model is a neural network architecture that reads all the text at once. It uses a mechanism called attention to decide which words matter for understanding each word.

Language models · transformer

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine a classroom where the teacher asks a question, and every student can look at all the classmates at once, not only at the one sitting next to them. That makes it easier to see who knows the answer and who needs help. A neural network built on this rule sees all the words of a sentence at the same time and notices which ones belong together.

In plain words

Older neural networks read text like a person reading aloud: word by word. By the end of a long paragraph, the start was forgotten. A transformer works differently. It looks at all the words at once and works out, for each one, which others deserve attention. In the sentence “The cat did not jump on the cupboard because it was too high”, the model links “it” to the cupboard. This scheme splits easily into many parallel calculations on a graphics card, so very large models could be trained on it. Almost all modern chatbots and translation tools use this design.

For business

Almost all language models that companies connect by API or install on their own servers use this architecture. Several practical points follow for a business. Cost and response time grow with the length of the request, because the model compares every token with every other token. The work needs graphics cards with a lot of memory; on ordinary processors, a large model answers too slowly. The upside of a shared architecture is that tools for running, compressing and fine-tuning models are interchangeable, so one developer’s model is easy to replace with another’s. Teams sort out these limits before they choose a server and a model, and they record them in the AI solution map.

For a specialist

Vaswani et al. proposed the Transformer in the paper “Attention Is All You Need” (2017). The architecture dropped recurrent and convolutional layers entirely in favor of attention. The original model has an encoder and a decoder. Each layer includes multi-head self-attention, a feed-forward block, residual connections and layer normalization. Positional encoding passes the order of the tokens. On the WMT 2014 English-French translation task, the model reached 41.8 BLEU after 3.5 days of training on eight GPUs. Three families followed: encoder-only models such as BERT (Devlin et al., 2018) for understanding text; decoder-only models such as GPT and most chat models for generation; and encoder-decoder models for converting text. Modern variants add rotary positional embeddings (RoPE), grouped-query attention, mixtures of experts and RMSNorm normalization.

Do not confuse. Not to be confused with an electrical transformer or with the Transformers robots from cartoons.

Example

An online translation tool gets the sentence “The bank approved the loan because it checked the income” and correctly understands that “it” refers to the bank. Attention inside the transformer makes this possible.

Related terms

Questions and answers

Why is the architecture called a transformer? The authors of the 2017 paper that first described the model do not explain the name. It is usually linked to the model’s task: it converts, or transforms, one sequence into another, for example a phrase in English into a phrase in Russian.
How is a transformer different from a recurrent neural network? A recurrent network reads text one word at a time and passes its memory along the chain, so it learns slowly and forgets the beginning. A transformer processes all the words in parallel and links any two of them directly through attention.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us