MMOOZZGG.com

Glossary

ROC AUC

In short

ROC AUC is the area under the ROC curve. It is a number from 0 to 1 that shows how well a model ranks positive examples above negative ones at any decision threshold.

Machine learning · ROC-AUC

MMOOZZGG.com editorsChecked: 2 min read

For a child

A teacher shows you pairs of pictures: one has a cat, the other a dog, and asks you to point to the cat. If you always get it right, you get full marks. If you point at random, you get about half of the pairs right. A computer is tested with a similar game, only with very many pairs. The more often it picks the right picture in a pair, the higher its score.

In plain words

Many models do not give a yes-or-no answer but a degree of confidence, such as the probability that a client will leave. Then you choose a threshold: above it, we assume the client will leave. The threshold can move. A low one catches almost everyone who leaves but raises many false alarms; a high one does the opposite. The ROC curve shows how the share of caught cases changes as the share of false alarms grows, across all thresholds at once. The area under this curve is the final score. A value of 1 means the model always ranks a leaving client above a staying one; 0.5 means a coin toss. The score does not depend on the chosen threshold, which is convenient.

For business

This metric helps compare scoring, churn and fraud detection models before a threshold is chosen. It answers the question of whether the model can tell risky clients from reliable ones at all. But in a working solution, money is counted at a specific threshold: how many clients you will call, how many payments you will hold. So after comparing by area, choose the threshold by the cost of errors and check the result with a confusion matrix. Another trap: with a rare event, such as one fraud case per thousand transactions, the metric can look high while the cases the model flags are almost all false alarms. In such tasks, also look at the precision-recall curve. Predictive analytics scenarios with this kind of check are discussed before development starts.

For a specialist

The ROC curve (receiver operating characteristic) is drawn across all decision thresholds as the true positive rate TPR = TP / (TP + FN) against the false positive rate FPR = FP / (FP + TN). The area under it (area under the curve, AUC) equals the probability that a randomly chosen positive object gets a higher score than a randomly chosen negative one. A value of 0.5 means random ranking and 1.0 perfect ranking; a value below 0.5 means that inverting the predictions gives a model better than random. The metric does not change under monotonic transformations of the scores and does not measure the calibration of probabilities. For multiclass tasks, scikit-learn supports one-vs-rest, which is sensitive to imbalance, and one-vs-one (Hand and Till, 2001). The max_fpr parameter gives a standardized partial area over the range of low FPR (McClish, 1989). With strong class imbalance, the precision-recall curve is more informative, as the Google machine learning crash course states directly.

Example

A bank compares two loan approval models and picks the one with the higher area under the ROC curve. Then a risk manager sets the threshold so that the share of overdue loans among approved applications stays within plan, and tests this threshold on the last quarter’s applications.

Related terms

Questions and answers

What ROC AUC value is considered good? A value of 0.5 means guessing, and 1.0 means perfect separation. The acceptable level depends on the task: in some fields a modest result is useful, in others you need an almost perfect model.

So compare the model with the current way of solving the task and with a baseline.

How is ROC AUC different from the F1 score? ROC AUC scores ranking across all thresholds at once. F1 is calculated at one chosen threshold and counts only the positive class.

The first metric is handy for comparing models, the second for checking a finished solution.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us