MMOOZZGG.com

Glossary

What is byte pair encoding (BPE)

In short

Byte pair encoding (BPE) is a tokenization algorithm. It builds a model’s vocabulary by merging the most frequent pairs of symbols into new units, again and again.

Language models · byte pair encoding

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you often write the word “mama” and are tired of writing every letter. You invent one sign for “ma”. Then you notice that “mama” appears all the time, so you invent a sign for the whole word. Step by step you get a set of signs for frequent pieces. A neural network builds its own set the same way.

In plain words

A neural network reads text in pieces called tokens, and something must decide what these pieces are. BPE decides it this way. It takes a huge set of texts and first splits everything into single letters. Then it finds the pair that appears most often, for example “s” and “t”, and declares it a new piece. It looks for the next frequent pair and repeats this thousands of times. As a result, frequent words become one token, and rare words are built from several parts, so the model can read any word, even an unknown one.

For business

The tokenizer is invisible, but it affects money. Models charge by tokens and limit request size in tokens, and how many tokens your text takes depends on the vocabulary BPE built. If the training texts had little Russian, Russian words are cut into small parts: the same document costs more and less of it fits in the context window. So when you choose a model, run typical company documents through its tokenizer and compare token counts. It also matters when you cost your own server: generation speed is measured in tokens per second, not words.

For a specialist

Byte Pair Encoding (BPE) was originally a data compression algorithm. Sennrich and coauthors (ACL, 2016) brought it into neural machine translation to encode rare and unknown words as sequences of subwords. Training: starting from characters, the algorithm repeatedly finds the most frequent pair of adjacent units in the corpus and adds that merge to the vocabulary until it reaches a set size; at tokenization the merges apply in the order they were learned. In GPT-2 (Radford and coauthors, 2019) BPE works on bytes: a base vocabulary of 256 values covers any UTF-8 text with no unknown tokens, and the final vocabulary has 50,257 units. The tiktoken library implements such a tokenizer; its description says encoding is reversible and lossless, and one token is about 4 bytes on average. Alternatives are WordPiece and the unigram language model in SentencePiece.

Example

A team compared how many tokens the same contract takes in two models. The model whose vocabulary was trained on more Russian text needed noticeably fewer tokens, and processing the whole archive cost less.

Related terms

Questions and answers

How does BPE work in plain words? The algorithm starts from single letters and again and again merges the pair that most often appears side by side. Each merge becomes a new piece of the vocabulary.

When the vocabulary reaches the set size, the process stops.

Why does Russian text take more tokens than English? The vocabulary is built from training texts, which usually hold more English. Frequent English words get their own tokens, while Russian words are more often cut into several pieces.

The exact ratio depends on the model.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us