MMOOZZGG.com

Glossary

What is a token in AI?

In short

A token is a piece of text that a language model works with: a word, part of a word, or a symbol.

Language models · token

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you build a phrase from letter blocks. A neural network also does not read text as a whole: it splits it into blocks. One block can be a whole word, or it can be half of a word. These blocks are called tokens.

In plain words

A neural network does not see words as we do. Before work, it cuts the text into pieces called tokens. A short, common word is usually one token. A long or rare word is split into several. The model reads tokens and then writes the answer one token at a time. So the length of a question and an answer is measured in tokens, not in words.

For business

Tokens in AI are the unit of both price and model memory. Cloud models charge for every million tokens of input and output, so long instructions and large documents cost more. A model’s context window is also counted in tokens: if a document does not fit, you need to shorten it or find the right parts with RAG. Russian text usually has more tokens for the same number of words than English text, and a budget should allow for that.

For a specialist

A token is an item in a tokenizer’s vocabulary, matched to an integer index. Most modern models use subword tokenization: variants of Byte Pair Encoding (BPE) and similar methods that merge frequent sequences of characters or bytes into one unit. The model gets a sequence of indices, turns them into embeddings and at each step predicts the probability distribution of the next token. The vocabulary size, the splitting rules and the share of texts in a given language in the training data decide how many tokens the same phrase takes.

Example

The phrase “The network answers the customer” may split into tokens like this: “The”, “ net”, “work”, “ answers”, “ the”, “ customer”. The exact split depends on the model.

Related terms

Questions and answers

How many tokens are in one word? It depends on the language and the model. A short, common word often takes one token, while a long Russian word may take three or four.

You can count exactly with the tokenizer of the model you use.

Why are AI answers paid for in tokens? The model spends compute on every token it reads and every token it writes. So cloud services set a price per million tokens, separately for input and output.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us