Glossary
What is perplexity in language models
In short
Perplexity is a quality metric for a language model that shows how “surprised” the model is by a text: the lower the value, the better the model predicts the next tokens.
Language models · perplexity
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a game: a friend reads a fairy tale, stops, and you guess the next word. If you know the tale, you almost always guess right, and there is hardly anything to choose from. If the tale is new to you, there are lots of options in your head. Scientists came up with a number that shows how many options a neural network has to choose from on average. The smaller it is, the more confidently the network guesses.
In plain words
To see how well a model knows a language, you give it a text it has not seen and look at the chances it gave to each real next word. If the chances were high, the model predicts the text well. The metric turns this into one number, similar to the average count of equally likely options the model hesitates between. A value of about ten means the model seems to choose from ten words, and about a thousand means it is almost lost. You can compare the numbers only on the same text.
For business
The metric is useful to developers, but it is a poor way to choose a model for a company’s task. A low value says the model predicts texts similar to the test texts well, and says nothing about whether it answers a customer correctly or extracts contract details. You cannot compare different models directly: the result depends on the tokenizer, and a model with a different vocabulary gets different numbers on the same text. Where the metric really helps: checking that quantization did not spoil a model for your own server, and comparing versions after fine-tuning on your documents. The final decision comes from tests on real tasks, which you run when you compare a local and a cloud option.
For a specialist
The perplexity (PPL) of an autoregressive model on a sequence X = (x_1, ..., x_t) equals the exponent of the average negative log-probability of the tokens: PPL = exp(-(1/t) Σ log p(x_i | x_<i)). It is the exponent of the cross-entropy between the data and the model, and it is read as the effective number of equally likely options per step. The Hugging Face documentation stresses that the metric is not well defined for masked models such as BERT and depends directly on tokenization, so models with different vocabularies are compared with care. For models with a limited window, the way you count matters: when WikiText-2 is split into non-overlapping chunks of 1024 tokens, GPT-2 large scores 19.44, while a sliding window with a stride of 512 lowers the score to 16.44, because each token has more context. The metric is used to monitor pretraining, assess quantization and filter training data, but it is only indirectly linked to the quality of answers on applied tasks.
Do not confuse. Do not confuse it with Perplexity, the AI search service: it has a separate page in this glossary.
Example
A team compresses an open model so that it fits on the company’s server. Perplexity on internal documents rose only slightly, and only after that is the model sent for a check with real questions from staff.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map