Glossary
What is a loss function in machine learning?
In short
A loss function in machine learning is a formula that turns a model’s error into one number: the smaller it is, the closer the model’s answers are to the right ones.
Machine learning · loss function
MMOOZZGG.com editorsChecked: 2 min read
For a child
Remember the game “hot or cold”? You look for a hidden toy, and a friend gives hints: “cold”, “warmer”, “very hot”. From these words you know where to go next. A computer that learns has such a helper too. It says how far the answer is from the right one, and it is called a loss function.
In plain words
While a model learns, it needs to know how far off it was. This formula compares the model’s answer with the correct one and gives a penalty: small for a small miss, large for a big one. Say a program valued an apartment at nine million, and it sold for ten. You can compute the penalty as the square of the difference. Then the penalties for all examples are added up, and training adjusts the settings to make the total as small as possible. A person chooses the formula, and that choice decides which misses the model treats as serious and which it forgives.
For business
In essence, this formula records what an “error” means for your company. If all misses cost the same, a scoring model may be wrong rarely on average but miss expensive defaults. So before training, agree with your contractor on the price of different errors: a missed fraudster, a false rejection of a good client, extra stock in the warehouse. These prices go into the class weights or into the formula itself. The main risk is a swapped goal: the model honestly lowers its penalty, but the penalty does not match the money you lose. Check that the metric in the report and the business figure are calculated consistently, and fix the price of errors in the task. Our map of AI solutions covers such questions.
For a specialist
A loss function sets the penalty for the gap between a prediction and the target value; training minimizes its average over the sample (empirical risk), often together with a regularization penalty. For regression, teams use the mean squared error (MSE), the mean absolute error (MAE) or the Huber loss: the PyTorch documentation says it is quadratic while the absolute error is below the delta threshold and linear above it, so it reacts less to outliers than MSE. For classification the standard is cross-entropy: CrossEntropyLoss in PyTorch takes unnormalized logits, can weight classes in an unbalanced sample and supports label smoothing. Gradient methods need a function that is differentiable almost everywhere, so step-like metrics such as the share of correct answers are replaced with convex surrogate losses. Backpropagation computes the gradient with respect to the network weights, and the optimizer takes the step.
Example
An online shop trained a demand forecast model with the mean squared error, and a few sales with huge volumes pulled the forecast up. After it switched to the Huber loss, these spikes had less effect, and purchasing stopped inflating orders.
Related terms
Questions and answers
How is a loss function different from a quality metric?
Which loss function should you choose?
Sources
Not sure whether your company needs this?
AI solutions map