Glossary
What is top-p sampling?
In short
Top-p sampling, also called nucleus sampling, is a generation setting in which a model picks the next token only among the most likely options whose total probability reaches a threshold p.
Language models · nucleus sampling
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine you are choosing ice cream at a cafe. If you really want chocolate or strawberry, you pick from your two or three favorites. If you do not mind, you look at most of the counter. But you never take strange flavors like mustard. A neural network picks words in a similar way: it throws out the ones that do not fit at all and draws from the rest.
In plain words
When a model writes an answer, at each step it has a list of possible next words with their chances. You can pick from the whole list, but then odd options from the long tail sometimes appear. Top-p cuts the tail. The options are sorted by chance and taken from the top until their sum reaches the threshold, for example ninety percent. If the model is sure, one or two words stay in the set. If it is unsure, the set grows. This keeps the text lively but stops it from turning into nonsense.
For business
The parameter is set in the API request or in the service settings. Like temperature, it costs nothing, but it affects how stable the answers are. For data extraction from documents, ticket classification and answers by company rules, lower it, so the model stays with the most likely wording. For ad copy and ideas, keep it high. A common mistake is to change it and temperature at the same time and then not know which one made the difference. It is easier to adjust one thing. The value is fixed in the agent’s configuration and tested on a set of typical requests before the agent meets live customers.
For a specialist
Holtzman et al. proposed nucleus sampling (2019, published at ICLR 2020). They showed that likelihood-maximizing decoding, including beam search, produces dull and repetitive text, while pure sampling from the full distribution pulls in an unreliable tail of unlikely tokens. The method takes the smallest set of tokens whose total probability is at least p, renormalizes the probabilities inside it and samples. Unlike top-k, the size of the set adapts: the authors observed that the nucleus usually holds from one to a thousand candidates, growing when the distribution is flat and shrinking when it is sharp. In Hugging Face Transformers, top_p defaults to 1.0, which means no cutoff, unless the model sets its own value in generation_config.json. A related filter, min-p, keeps tokens whose probability is at least a given share of the probability of the most likely token.
Example
A service generates product descriptions. With top-p close to one, the texts sometimes contain strange words. After the threshold is lowered, the descriptions read more evenly but stay different for similar products.
Related terms
Questions and answers
How is top-p different from temperature?
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map