MMOOZZGG.com

Glossary

What is class imbalance in machine learning?

In short

Class imbalance in machine learning is a situation where the training data has far more examples of one class than of another, and the model almost stops noticing the rare class.

Machine learning · class imbalance

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine a box with a hundred red balls and only two blue ones. If you always say “red”, you will almost never be wrong, but you will never learn to find the blue ones. A computer that learns from such examples does something similar: it gets used to the common answer. So people make sure the rare examples are easy to see too.

In plain words

A model learns from examples and notices which answers come up more often. If only a few of a thousand payments are fraud, it pays for the model to always answer “the payment is honest”: the share of correct answers is almost perfect, and the use is zero. This skew is fixed in three ways: show the rare examples more often, thin out the common ones, or tell the model directly that missing a rare case costs more. It is like a doctor who for years saw only healthy patients: a rare disease is easy for them to miss.

For business

Almost every valuable task has this skew: fraud, defects on a line, customer churn, equipment failure. The interesting event is always rare, and it is exactly the one the company wants to catch. The danger hides in reports: a contractor shows an impressive share of correct answers, while the model has not found a single fraudster. Before you accept the work, ask for recall and precision on the rare class and a confusion matrix on a real sample. The cost of the solution depends on how many rare cases have built up in the history and what a miss costs compared with a false alarm. Sometimes it is more honest to collect data for a few months first than to train a model on a dozen examples. Check such assumptions in a pilot on a live flow of operations.

For a specialist

Class imbalance is an uneven distribution of labels in a classification task, where the minority class is a small share of the sample. The loss function averaged over examples is then driven mostly by the majority class, and accuracy loses its meaning; quality is measured by precision and recall, the F1 score, PR-AUC and the confusion matrix. Methods fall into three groups. Data level: random undersampling, oversampling of rare examples and synthesis of new ones; SMOTE (Chawla et al., JAIR, 2002) creates synthetic minority points between neighboring examples and is combined with undersampling of the majority class. Loss level: class weights, cost-sensitive learning, focal loss (Lin et al., 2017), which lowers the weight of well-classified examples and focuses training on hard ones. Decision level: threshold shift and probability calibration. A stratified split keeps the share of classes in the training and test parts.

Example

A bank trains a model to find fraudulent transfers, and in its history there are only a handful of them among many thousands of honest ones. Without a fix for the skew, the model missed almost all suspicious operations, though it looked accurate in the report. After class weighting and a threshold shift, it began to raise alarms more often, and staff now review only the flagged transfers.

Related terms

Questions and answers

Why can’t you look at the share of correct answers when classes are imbalanced? If the rare class is a small part of the data, a model that always picks the common answer gets an almost perfect share of correct answers. Yet it is of no use.

So you look at recall and precision for the rare class itself.

How do you fix class imbalance? There are three ways: change the data, that is, repeat or synthesize rare examples and thin out common ones; give the rare class a larger weight in the loss function; or move the decision threshold. You check which way fits on a held-out set with the real ratio of classes.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us