Glossary
What is tokenization?
In short
Tokenization is the splitting of text into tokens: pieces of words and symbols that a language model turns into numbers and only then reads.
Language models · tokenization
MMOOZZGG.com editorsChecked: 1 min read
For a child
When you learn to read, you split a long word into syllables: “com-pu-ter”. A computer also finds it hard to understand a whole phrase at once, so a special program cuts it into small pieces in advance. It gives each piece its own number, like a card in a bingo box. After that, the computer works with numbers, not letters.
In plain words
A model cannot read letters. It works with numbers. So before every request, the text passes through a tokenizer, a program with a dictionary of text pieces. It finds known pieces in the text and replaces each with its number. Frequent words sit in the dictionary whole, while rare ones are built from several parts. For example, the word “contract” may stay one piece, while a client’s unusual surname falls apart into three or four. On the way out, everything runs in reverse: numbers are joined back into words, and you see a normal answer.
For business
Tokenization shapes the bill for a cloud model. Prices are counted in tokens, and the same text is cut into a different number of pieces by different models. Russian text often takes more tokens than English text with the same meaning if the model’s vocabulary has few Russian fragments. So comparing providers by their price list alone is not accurate: run your typical documents through each one and count the real usage. Tokenization also decides how many pages fit into one request and how fast the answer comes. This measurement fits well into a pilot before you choose a provider.
For a specialist
Tokenization turns a string into a sequence of integer indices from a fixed vocabulary. Word-level tokenizers stumble on unknown words, and character-level ones produce sequences that are too long, so language models settled on the subword approach. Sennrich, Haddow and Birch (ACL, 2016) applied Byte Pair Encoding (BPE) to translation. The algorithm starts with single characters and repeatedly merges the most frequent pair into a new unit until the vocabulary reaches a set size. Kudo and Richardson (2018) released SentencePiece. It trains directly on raw sentences without splitting them into words first and supports BPE and the unigram language model (unigram LM). Byte-level BPE works with UTF-8 bytes and has no unknown symbols. Normalization, pre-tokenization and special tokens (BOS, EOS) are part of the pipeline and must match in training and inference.
Example
A company compared two cloud models on a hundred real customer emails. By the price list, the first was cheaper. But its vocabulary cut Russian text into many more pieces, and the final bill turned out higher.
Related terms
Questions and answers
Why does Russian text cost more in tokens?
Sources
Not sure whether your company needs this?
AI solutions map