MMOOZZGG.com

Glossary

What is top-p sampling?

In short

Top-p sampling, also called nucleus sampling, is a generation setting in which a model picks the next token only among the most likely options whose total probability reaches a threshold p.

Language models · nucleus sampling

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you are choosing ice cream at a cafe. If you really want chocolate or strawberry, you pick from your two or three favorites. If you do not mind, you look at most of the counter. But you never take strange flavors like mustard. A neural network picks words in a similar way: it throws out the ones that do not fit at all and draws from the rest.

In plain words

When a model writes an answer, at each step it has a list of possible next words with their chances. You can pick from the whole list, but then odd options from the long tail sometimes appear. Top-p cuts the tail. The options are sorted by chance and taken from the top until their sum reaches the threshold, for example ninety percent. If the model is sure, one or two words stay in the set. If it is unsure, the set grows. This keeps the text lively but stops it from turning into nonsense.

For business

The parameter is set in the API request or in the service settings. Like temperature, it costs nothing, but it affects how stable the answers are. For data extraction from documents, ticket classification and answers by company rules, lower it, so the model stays with the most likely wording. For ad copy and ideas, keep it high. A common mistake is to change it and temperature at the same time and then not know which one made the difference. It is easier to adjust one thing. The value is fixed in the agent’s configuration and tested on a set of typical requests before the agent meets live customers.

For a specialist

Holtzman et al. proposed nucleus sampling (2019, published at ICLR 2020). They showed that likelihood-maximizing decoding, including beam search, produces dull and repetitive text, while pure sampling from the full distribution pulls in an unreliable tail of unlikely tokens. The method takes the smallest set of tokens whose total probability is at least p, renormalizes the probabilities inside it and samples. Unlike top-k, the size of the set adapts: the authors observed that the nucleus usually holds from one to a thousand candidates, growing when the distribution is flat and shrinking when it is sharp. In Hugging Face Transformers, top_p defaults to 1.0, which means no cutoff, unless the model sets its own value in generation_config.json. A related filter, min-p, keeps tokens whose probability is at least a given share of the probability of the most likely token.

Example

A service generates product descriptions. With top-p close to one, the texts sometimes contain strange words. After the threshold is lowered, the descriptions read more evenly but stay different for similar products.

Related terms

Questions and answers

How is top-p different from temperature? Temperature changes the shape of the whole distribution: it makes the choice bolder or more careful. Top-p cuts off the unlikely tail and keeps only a set of plausible options.

Usually you adjust one of the two parameters, not both at once.

What top-p value should you set? For precise tasks, lower the threshold; for creative ones, keep it close to one. Pick the exact value by testing on your own requests, because default settings differ between models and services.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us