MMOOZZGG.com

Glossary

What is LLM temperature?

In short

LLM temperature is a generation setting that controls how random a language model’s answer is: with a low value the answers are more predictable, with a high value more varied.

Language models · temperature

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine you are choosing a cartoon to watch. On a calm day you put on your favorite, the one you have seen a hundred times. On a fun day you might pick something totally unexpected. A neural network has a similar mood switch: in the calm position it answers in the usual way, and in the fun position it invents unusual words and ideas.

In plain words

A model writes an answer piece by piece. At each step it picks the next word from a list of fitting options, and each option has its own probability. This setting changes the odds. At a low value the model almost always takes the most likely option and gives the same answer to the same question every time. At a high value rare options get more chances: the answers become livelier and more varied, but drift off topic more often. For example, you raise the value to invent names for a new coffee shop and lower it almost to zero to extract company details from an invoice.

For business

You set temperature in the service settings or in an API request. It costs nothing but has a visible effect on quality. For tasks with one right answer, such as sorting requests, extracting data from documents and answering by internal rules, set it low so the result is stable and repeatable. For ad drafts, ideas and email variants, raise it. A wrong setting looks like a model problem: a support agent with a high value sometimes invents terms that are not in the knowledge base. A low value does not remove made-up answers completely. Fix the setting in the configuration and run a test set of questions several times in a row; this check is part of a pilot.

For a specialist

Temperature T divides the model’s logits before the softmax function: p_i = exp(z_i / T) / Σ exp(z_j / T). With T below 1 the distribution becomes sharper; as T approaches zero, the choice turns into greedy decoding; with T above 1 the distribution becomes flatter and the share of unlikely tokens grows. In the Hugging Face Transformers library the default value is 1.0 unless the model sets its own in generation_config.json, and the do_sample flag turns sampling on. The setting is combined with truncation of the distribution: top-k keeps the k most likely tokens, and top-p (nucleus sampling), by Holtzman and co-authors (2019), keeps the smallest set of tokens whose total probability is at least p. The authors showed that decoding by maximum likelihood gives dull, repetitive text, while cutting the unreliable tail keeps the text coherent and varied. A value of zero does not give full reproducibility because GPU computing is not deterministic.

Example

A marketer asks the model for twenty names for a promotion and sets a high temperature so the options do not repeat. An accountant uses the same model with a value near zero to sort invoices, so the amounts come out the same on every run.

Related terms

Questions and answers

What is temperature in neural networks? It is a setting that decides how randomly the model picks the next word. A low value gives precise and uniform answers.

A high value makes them more varied but raises the risk of errors.

What value should you set for work tasks? For data extraction, classification and answers from documents, set a low value close to zero. For ideas and creative text, raise it.

Find the exact value by testing on your own tasks, because the scales differ between services.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us