Glossary
What is the attention mechanism?
In short
The attention mechanism is the way a neural network decides, for each word, which other words of the text to rely on and how strongly.
Language models · attention
MMOOZZGG.com editorsChecked: 1 min read
For a child
When you read a fairy tale and see the word “he”, you look back to find who it is about: the wolf or the hare. It is as if you shine a flashlight on the right place on the page. A neural network does something similar: for each word, it shines a flashlight on the other words and looks at which ones help it understand the meaning.
In plain words
The meaning of a word often depends on its neighbors. The word “bank” in “river bank” and “bank loan” means different things. Attention helps the model tell them apart. For each word, the model rates all the other words in the text and gives them scores: the more important a neighbor is for understanding, the higher its score. Then the model mixes the information from the neighbors, weighted by the scores, and the word gets a description that includes its surroundings. This repeats many times on different layers, and step by step the model learns who did what and which noun a pronoun refers to.
For business
Attention explains why long requests are expensive. Each new word is compared with all the earlier words, so the computation grows faster than the length of the text, and the memory for intermediate data grows with the length of the request. For a company, this means you can technically load a whole archive of contracts into one request, but the answer will be slow and costly, and on your own server you will need more video memory. Optimized versions of attention cut the cost, so check whether your serving software supports them. Attention maps sometimes hint at which parts of a document the model relied on, but they are not a full explanation of its decision. Plan memory for long documents when you choose a server.
For a specialist
The attention mechanism computes, for each query, a weighted sum of values, where the weights come from the similarity of the query to the keys. It was first used in neural machine translation by Bahdanau, Cho and Bengio (2014): at each step the decoder softly searched for the relevant parts of the source sentence instead of compressing it into one fixed-length vector. The transformer (Vaswani et al., 2017) uses scaled dot-product attention: softmax(QK^T / sqrt(d_k)) V. Self-attention takes Q, K and V from one sequence, cross-attention links the decoder with the encoder, and multi-head attention computes several projections in parallel. The cost is quadratic in the sequence length, hence sparse, linear and sliding-window variants. FlashAttention (Dao et al., 2022) computes exact attention in blocks and reduces data movement between slow and fast GPU memory.
Example
An AI support agent gets the message “I ordered two phone cases, one arrived damaged, can I return it?”. Thanks to attention, it understands that “it” means the damaged case, not the whole order.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map