MMOOZZGG.com

Glossary

What is top-k sampling?

In short

Top-k sampling is a generation setting in which a language model picks the next token only from the k most likely options at each step.

Language models · top-k sampling

MMOOZZGG.com editorsChecked: 1 min read

For a child

A teacher asks the class a question, and twenty students raise their hands. She decides: I will pick one, but only from the first five who raised a hand fastest. So the person who answers is surely ready, but it is not always the same student. A neural network picks a word in a similar way: from a short list of good options, skipping the rest.

In plain words

At each step, the model scores every word it knows, and each word has a chance to come next. Top-k says: consider only the k options with the highest chances and cross out the rest. If k is one, the model always takes the most likely word and answers the same way each time. If k is large, the choice is wider and the text is more varied. For example, with k around ten, a request for a slogan gets a new answer each time, but without senseless words from the tail of the list.

For business

Like other word-choice settings, k costs nothing and is set in the request or in the server settings. Keep in mind that a fixed number of options works differently in different parts of an answer. Where there is only one correct continuation, such as a contract number or an amount, even a small k leaves a chance of picking a wrong token if sampling is on. Where many options fit, a k that is too small makes the text formulaic. So for tasks with an exact answer, sampling is usually turned off, and for free text, k is tuned together with temperature. Whether the parameter is available depends on the service, so check your provider’s documentation. Generation settings for a support agent are tested on real customer requests.

For a specialist

Fan, Lewis and Dauphin (2018) used top-k sampling for story generation: the model picked a word at random from the 10 most likely candidates. The authors found this strategy much more effective than beam search, which produces common phrases and repetition. The algorithm sorts the distribution, keeps the k tokens with the highest probabilities, sets the rest to zero and renormalizes. With k = 1, you get greedy decoding. Holtzman et al. (2019) analyzed the weakness of a fixed k: with a flat distribution, it cuts off fitting options, and with a sharp one, it keeps unfitting ones. So they proposed adaptive nucleus sampling (top-p). In practice, filters are combined: in Hugging Face Transformers, top_k defaults to 50 unless the model sets its own value, and it works together with top_p and temperature when do_sample=True. Technically, the filter is applied to logits: values outside the set are replaced with minus infinity before softmax.

Example

The developer of a coffee shop chatbot sets top-k to forty, and the bot greets guests in a new way each time. To calculate the order total, the same bot makes a separate call without sampling, so the numbers do not change.

Related terms

Questions and answers

How is top-k different from top-p? Top-k always keeps the same number of options. Top-p keeps as many as needed to reach a set share of probability: few options when the model is sure, more when it is unsure.
What happens if you set top-k to one? The model will always take the most likely token. This is greedy decoding.

Answers become the same on repeated runs, but long texts show more repetition.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us