MMOOZZGG.com

Glossary

What is positional encoding

In short

Positional encoding is a way to tell a transformer the order of words: it adds information about the place of each token in the text to the description of that token.

Language models · positional encoding

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you pour cards with the words of a poem onto a table. The words are the same, but the poem is gone because the order is mixed up. To put it back together, you write the line number and the place in the line on each card. A neural network needs such numbers too. Without them it sees the words as a pile and does not know what comes after what.

In plain words

A transformer looks at all the words of a text at the same time. This is fast, but the scheme has no built-in sense of order. For it, “payment first, then delivery” and “delivery first, then payment” are made of the same set of pieces. So each token gets a place mark mixed into it. In the first models this was a set of waves of different frequencies, with its own pattern for each position. In modern models the mark is more often built right into the attention calculation, and the model takes into account how far apart the words stand.

For business

Companies rarely choose a positional encoding scheme directly, but it decides something that matters to them: how much text the model reads reliably at once. The scheme decides whether the model can handle a document longer than the ones it was trained on. Some schemes let you stretch the context window after training, but quality on distant parts may drop: the model starts to miss terms from the middle of a long contract. So do not take the stated window size on trust. Before rollout, run the model on your own long documents and check that it finds facts from the beginning, middle and end. This test is part of the pilot.

For a specialist

Positional encoding adds order information to an architecture that otherwise cannot tell permutations of tokens apart. Vaswani and co-authors (2017) added sine and cosine waves of different frequencies to the embeddings, and learned positional embeddings gave almost the same result in their work. These are absolute schemes: a position is set by a number. Relative schemes use the distance between tokens. RoPE (Rotary Position Embedding, Su and co-authors, 2021) rotates the query and key vectors by a rotation matrix whose angle depends on position, so their dot product in attention depends on the difference of positions. ALiBi (Press, Smith and Lewis, 2021) drops positional embeddings and penalizes attention scores in proportion to distance. A 1.3-billion-parameter model trained on sequences of 1,024 tokens reached, at 2,048 tokens, the same perplexity as a sinusoidal model trained directly on 2,048. To extend the context window of RoPE models, teams use position interpolation.

Example

A lawyer uploads a contract of one hundred pages and asks about a penalty from the middle of the text. If the position scheme works poorly at this length, the model confidently quotes a similar clause from the beginning and gets it wrong.

Related terms

Questions and answers

Why does a transformer need positional encoding? Attention alone does not know in which order the tokens stand. A position mark lets the model tell “the cat caught the mouse” from the same words in another order and see which word is closer to which.
How is RoPE different from sinusoidal encoding? Sinusoidal encoding adds a mark of its number to each token. RoPE rotates the query and key vectors, so attention depends on the distance between tokens, not on their absolute numbers.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us