MMOOZZGG.com

Glossary

What is data drift?

In short

Data drift is a change in real-world data after a model goes live, so that the data no longer looks like the training data and the quality of predictions falls quietly.

Machine learning · data drift

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine that you learned to recognize everyone in your class. Over the summer they grew, someone got a haircut, someone started wearing glasses. In the autumn you mix up names, though your memory is the same. The world around you just changed. The same happens with a computer program, so from time to time it is taught again on fresh examples.

In plain words

A model learns from the past and silently assumes that the future looks like it. But life changes: customers get new habits, a store gets a new range of goods, a sensor on a machine is replaced with another one. Incoming data slowly moves away from the data the model learned on, and the answers get worse. The error is not visible at once: the system keeps giving confident predictions. Example: a demand model trained before a price rise still expects the old sales volumes, and the warehouse buys too much.

For business

For a company, this is a common reason why a model that passed its pilot starts making mistakes six months later. Sources: seasons, new products, changes in laws and tariffs, new equipment or a new export format from the accounting system. The problem is costly precisely because it is quiet: reports keep being built, but the decisions based on them are already wrong. What to plan before launch: a regular comparison of incoming data with the training data, quality control on fresh labeled examples, a threshold after which the model is retrained, and a person responsible for these signals. The cost is monitoring plus periodic retraining; put it into the support budget, not only the adoption budget. Write the support procedure into the contract.

For a specialist

Data drift (dataset shift) is a change in the joint distribution P(X, Y) between training and production. There is covariate shift, where P(X) changes while P(Y|X) stays the same; label shift, where P(Y) changes while P(X|Y) stays the same; and concept drift, where the relationship P(Y|X) itself changes. By the nature of the change, drift is sudden, gradual, incremental or recurring. The survey by Lu et al. (IEEE TKDE, 2018) splits the work with drift into three tasks: detection, understanding and adaptation. For detection, data windows are compared with statistical tests (Kolmogorov-Smirnov, chi-square), the population stability index (PSI) and detectors on the error stream (DDM, ADWIN). Rabanser, Günnemann and Lipton (NeurIPS 2019) compared methods and found that two-sample tests after dimensionality reduction with a pretrained classifier work most reliably. Adaptation: retraining on a schedule or on a signal, a sliding window, example weighting, online learning.

Example

A pharmacy chain trained a demand forecast model on data from two calm years. In the autumn a wave of colds came, and the model underestimated demand for fever medicine. Monitoring spotted the gap with actual sales within a week, and the model was retrained on fresh sales.

Related terms

Questions and answers

How do you detect data drift? Compare the distributions of incoming data with the training data: means, shares of categories, statistical tests, the stability index. At the same time, check quality on fresh labeled examples, because not every change in the input spoils the answers.
What should you do if a model starts to make mistakes because the data changed? First find the cause: an error in the data export, the season or a real change in the market. Then fine-tune or retrain the model on fresh data.

If the changes repeat, set up training on a schedule or on a monitoring signal.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us