MMOOZZGG.com

Glossary

What is a loss function in machine learning?

In short

A loss function in machine learning is a formula that turns a model’s error into one number: the smaller it is, the closer the model’s answers are to the right ones.

Machine learning · loss function

MMOOZZGG.com editorsChecked: 2 min read

For a child

Remember the game “hot or cold”? You look for a hidden toy, and a friend gives hints: “cold”, “warmer”, “very hot”. From these words you know where to go next. A computer that learns has such a helper too. It says how far the answer is from the right one, and it is called a loss function.

In plain words

While a model learns, it needs to know how far off it was. This formula compares the model’s answer with the correct one and gives a penalty: small for a small miss, large for a big one. Say a program valued an apartment at nine million, and it sold for ten. You can compute the penalty as the square of the difference. Then the penalties for all examples are added up, and training adjusts the settings to make the total as small as possible. A person chooses the formula, and that choice decides which misses the model treats as serious and which it forgives.

For business

In essence, this formula records what an “error” means for your company. If all misses cost the same, a scoring model may be wrong rarely on average but miss expensive defaults. So before training, agree with your contractor on the price of different errors: a missed fraudster, a false rejection of a good client, extra stock in the warehouse. These prices go into the class weights or into the formula itself. The main risk is a swapped goal: the model honestly lowers its penalty, but the penalty does not match the money you lose. Check that the metric in the report and the business figure are calculated consistently, and fix the price of errors in the task. Our map of AI solutions covers such questions.

For a specialist

A loss function sets the penalty for the gap between a prediction and the target value; training minimizes its average over the sample (empirical risk), often together with a regularization penalty. For regression, teams use the mean squared error (MSE), the mean absolute error (MAE) or the Huber loss: the PyTorch documentation says it is quadratic while the absolute error is below the delta threshold and linear above it, so it reacts less to outliers than MSE. For classification the standard is cross-entropy: CrossEntropyLoss in PyTorch takes unnormalized logits, can weight classes in an unbalanced sample and supports label smoothing. Gradient methods need a function that is differentiable almost everywhere, so step-like metrics such as the share of correct answers are replaced with convex surrogate losses. Backpropagation computes the gradient with respect to the network weights, and the optimizer takes the step.

Example

An online shop trained a demand forecast model with the mean squared error, and a few sales with huge volumes pulled the forecast up. After it switched to the Huber loss, these spikes had less effect, and purchasing stopped inflating orders.

Related terms

Questions and answers

How is a loss function different from a quality metric? The training algorithm needs the loss: the formula is smooth, and it is easy to compute weight updates from it. People need the metric: the share of correct answers, precision and recall, money.

They are related but not the same, so you train a model on the loss and accept it on the metric.

Which loss function should you choose? For numbers, teams usually take the mean squared or mean absolute error; for classes, cross-entropy. If the data has many outliers, the Huber loss fits.

If errors cost different amounts, give the classes different weights.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us