Glossary
What is a transformer model?
In short
A transformer model is a neural network architecture that reads all the text at once. It uses a mechanism called attention to decide which words matter for understanding each word.
Language models · transformer
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine a classroom where the teacher asks a question, and every student can look at all the classmates at once, not only at the one sitting next to them. That makes it easier to see who knows the answer and who needs help. A neural network built on this rule sees all the words of a sentence at the same time and notices which ones belong together.
In plain words
Older neural networks read text like a person reading aloud: word by word. By the end of a long paragraph, the start was forgotten. A transformer works differently. It looks at all the words at once and works out, for each one, which others deserve attention. In the sentence “The cat did not jump on the cupboard because it was too high”, the model links “it” to the cupboard. This scheme splits easily into many parallel calculations on a graphics card, so very large models could be trained on it. Almost all modern chatbots and translation tools use this design.
For business
Almost all language models that companies connect by API or install on their own servers use this architecture. Several practical points follow for a business. Cost and response time grow with the length of the request, because the model compares every token with every other token. The work needs graphics cards with a lot of memory; on ordinary processors, a large model answers too slowly. The upside of a shared architecture is that tools for running, compressing and fine-tuning models are interchangeable, so one developer’s model is easy to replace with another’s. Teams sort out these limits before they choose a server and a model, and they record them in the AI solution map.
For a specialist
Vaswani et al. proposed the Transformer in the paper “Attention Is All You Need” (2017). The architecture dropped recurrent and convolutional layers entirely in favor of attention. The original model has an encoder and a decoder. Each layer includes multi-head self-attention, a feed-forward block, residual connections and layer normalization. Positional encoding passes the order of the tokens. On the WMT 2014 English-French translation task, the model reached 41.8 BLEU after 3.5 days of training on eight GPUs. Three families followed: encoder-only models such as BERT (Devlin et al., 2018) for understanding text; decoder-only models such as GPT and most chat models for generation; and encoder-decoder models for converting text. Modern variants add rotary positional embeddings (RoPE), grouped-query attention, mixtures of experts and RMSNorm normalization.
Do not confuse. Not to be confused with an electrical transformer or with the Transformers robots from cartoons.
Example
An online translation tool gets the sentence “The bank approved the loan because it checked the income” and correctly understands that “it” refers to the bank. Attention inside the transformer makes this possible.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map