MMOOZZGG.com

Glossary

What is Attention Is All You Need?

In short

Attention Is All You Need is a 2017 scientific paper in which eight authors proposed the transformer architecture, which became the base of modern language models.

History of AI · Attention Is All You Need

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you read a fairy tale. When you see the word “she”, you know at once it means the princess, not the carriage. Computers used to read line by line and forget the beginning. A few scientists wrote a paper that suggested looking at all the words at once and noticing which ones are linked. This idea helped build assistants that write and translate texts.

In plain words

It is the title of a scientific paper, not a program. In 2017, researchers proposed building a translation model entirely from an attention mechanism: each word decides which words of the phrase to look at. Before that, texts were handled by recurrent networks, which read one word at a time and split work poorly between processors. The new design, the transformer, trained faster and translated more accurately. A few years later, models built on it began to answer in chats, write code and summarize documents. That is why the title is famous: almost every modern chatbot traces its family tree back to it.

For business

For a manager, the paper matters for its effects, not its formulas. The design it proposed became the common industry standard, so models from different developers are built in similar ways, and the software to run and tune them fits many of them. That means less dependence on one vendor: you can switch models without rewriting the whole project. The limits come from the same paper. Attention compares every word with every other word, so long documents take longer and cost more, and your own server needs graphics cards with large memory. These facts matter when you decide where to run a model: in the cloud or on your own hardware.

For a specialist

The paper by Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin was posted on arXiv on 12 June 2017 and published in the NIPS 2017 proceedings (Advances in Neural Information Processing Systems 30). The authors dropped recurrence and convolutions: the encoder and decoder consist of multi-head self-attention layers and position-wise feed-forward blocks, and positional encoding sets the word order. With no sequential pass over tokens, training parallelizes well on GPUs. On the WMT 2014 English-German task, the transformer scored 28.4 BLEU, more than 2 points above earlier results, including ensembles. This architecture led to encoder-only models such as BERT and decoder-only models such as GPT, and then to large language models.

Example

An engineer chooses a model for contract search and sees “decoder-only transformer” in its description. He knows it descends from the architecture of the 2017 paper, and checks at once how long a document the model accepts and how much memory it will take on the server.

Related terms

Questions and answers

Who wrote Attention Is All You Need? Eight researchers: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser and Illia Polosukhin. The paper appeared on arXiv in June 2017.
What is the transformer in this paper? It is a neural network architecture where an attention mechanism computes the links between words, with no recurrent or convolutional layers. Almost all modern language models are built on it.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us