MMOOZZGG.com

Glossary

Softmax function: what it is

In short

The softmax function turns a set of any numbers into probabilities: every value becomes a number between zero and one, and all values add up to one.

Language models · softmax

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you and your friends share a big cake, and each of you said in advance how much you want something sweet. Whoever wants it more gets a bigger piece, but the whole cake is shared with nothing left over, not a single extra crumb. A neural network splits its confidence between answer options in exactly the same way: all of it, by the strength of the wish.

In plain words

At the output, a model gives each option a plain number, sometimes a negative one. People and programs need clear chances. This function does the conversion: it raises e to the power of each number so they all become positive, then divides by the total. The result is shares that add up to one. After the conversion, large numbers pull even further ahead of small ones, so a clear favorite takes almost all the chance. This is how scores for email categories become clear shares for “complaint”, “question” and “thanks” that an operator can read.

For business

Softmax sits inside almost every neural network that has to choose: the next word, a request category, a document topic. For an executive one consequence matters: the output probabilities always add up to one hundred percent, even when no option fits. A model trained on three categories will put an email about something new into one of them and may show high confidence. That is why classification systems add an “other” category and a threshold below which a person decides. The second practical point: the temperature setting works through this very function, flattening or sharpening the distribution. You can see where such thresholds are needed in processes when you look at how AI works inside a company.

For a specialist

The softmax function maps a vector z from R^n onto the probability simplex: σ(z)_i = exp(z_i) / Σ_j exp(z_j). It is differentiable, monotonic in each argument and does not change when all components shift by a constant, so the maximum is subtracted from the logits to avoid overflow of the exponent. With temperature T the arguments are divided by T: as T approaches zero the function turns into argmax, and as T grows the distribution approaches uniform. Paired with cross-entropy, the gradient with respect to the logits equals the difference between the predicted probabilities and the one-hot target, which simplifies training of classifiers. The PyTorch documentation recommends LogSoftmax for training instead of Softmax followed by a logarithm, as a faster and numerically stable option. In the transformer of Vaswani and co-authors (2017), softmax normalizes the attention scores QK^T divided by the square root of d_k. Hierarchical and adaptive versions speed it up on large vocabularies, and sparsemax gives sparse distributions.

Example

A model sorts requests into “return”, “delivery” and “payment”. A customer asks about a job opening, and softmax still splits the probability among the three topics, with one of them getting most of it. So the team added a threshold and a manual check to the system.

Related terms

Questions and answers

Why do you need the softmax function? It turns the model’s raw scores into probabilities that are easy to work with: pick an option, compare it with a threshold, compute the error during training. Without it the output numbers could not be read as chances.
Why can softmax be confident in a wrong answer? The function always splits all the probability among the known options. If the right one is not among them, one of the wrong ones still gets a large share, so you need a threshold and an “other” option.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us