MMOOZZGG.com

Glossary

What is tokenization?

In short

Tokenization is the splitting of text into tokens: pieces of words and symbols that a language model turns into numbers and only then reads.

Language models · tokenization

MMOOZZGG.com editorsChecked: 1 min read

For a child

When you learn to read, you split a long word into syllables: “com-pu-ter”. A computer also finds it hard to understand a whole phrase at once, so a special program cuts it into small pieces in advance. It gives each piece its own number, like a card in a bingo box. After that, the computer works with numbers, not letters.

In plain words

A model cannot read letters. It works with numbers. So before every request, the text passes through a tokenizer, a program with a dictionary of text pieces. It finds known pieces in the text and replaces each with its number. Frequent words sit in the dictionary whole, while rare ones are built from several parts. For example, the word “contract” may stay one piece, while a client’s unusual surname falls apart into three or four. On the way out, everything runs in reverse: numbers are joined back into words, and you see a normal answer.

For business

Tokenization shapes the bill for a cloud model. Prices are counted in tokens, and the same text is cut into a different number of pieces by different models. Russian text often takes more tokens than English text with the same meaning if the model’s vocabulary has few Russian fragments. So comparing providers by their price list alone is not accurate: run your typical documents through each one and count the real usage. Tokenization also decides how many pages fit into one request and how fast the answer comes. This measurement fits well into a pilot before you choose a provider.

For a specialist

Tokenization turns a string into a sequence of integer indices from a fixed vocabulary. Word-level tokenizers stumble on unknown words, and character-level ones produce sequences that are too long, so language models settled on the subword approach. Sennrich, Haddow and Birch (ACL, 2016) applied Byte Pair Encoding (BPE) to translation. The algorithm starts with single characters and repeatedly merges the most frequent pair into a new unit until the vocabulary reaches a set size. Kudo and Richardson (2018) released SentencePiece. It trains directly on raw sentences without splitting them into words first and supports BPE and the unigram language model (unigram LM). Byte-level BPE works with UTF-8 bytes and has no unknown symbols. Normalization, pre-tokenization and special tokens (BOS, EOS) are part of the pipeline and must match in training and inference.

Example

A company compared two cloud models on a hundred real customer emails. By the price list, the first was cheaper. But its vocabulary cut Russian text into many more pieces, and the final bill turned out higher.

Related terms

Questions and answers

How is tokenization different from a token? A token is a single piece of text, and tokenization is the process that cuts text into such pieces. The tokenizer of each model sets the rules.
Why does Russian text cost more in tokens? A tokenizer’s vocabulary is built from training texts. If they contain less Russian text, Russian words are split into several parts more often, and the same meaning takes more tokens.

The same is true for other languages that are rare in the training data.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us