Glossary
What is stacking in machine learning?
In short
Stacking in machine learning is a way to combine several models: their predictions become the input for one more model, and that model gives the final answer.
Machine learning · stacking
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a quiz where three friends answer for you. One knows a lot about animals, the second about space, the third about sports. At first you just listen to all of them. Then you notice who is right more often on which questions, and you learn to choose whose answer to trust. You do not guess anything yourself; you become the judge of your friends’ hints. A computer program can also become such a judge over other programs.
In plain words
First you train several different models, say a linear model, a tree and a neural network. Each gives its own prediction. Then these predictions go into a second model, called a meta-model. It learns in which situations to trust which of the first models and adds up their answers with suitable weights. An important detail: the meta-model learns from predictions the first models made for examples they did not see in training. Otherwise it will trust the model that simply memorized the answers. It is like a chief editor who has known the authors for years: one is trusted on numbers, another on wording.
For business
The method helps when a team already has several working models and wants a bit more quality from them, for example in scoring, churn forecasts or payment default risk. The gain is usually modest, while complexity grows noticeably: several models must be trained, updated and monitored, answers take longer to compute, and it is harder to explain a decision to a client or a regulator. A mistake in splitting the data gives a false success that disappears in production. So before rollout, compare the stack with a single model on a fresh period and decide whether the gain is worth the support cost. If the use case is on the AI solutions map, this calculation is done before you order development.
For a specialist
Stacking (stacked generalization) was described by D. H. Wolpert in the journal Neural Networks in 1992. First-level (level-0) models train on the original features, and their outputs, class probabilities or numeric predictions, serve as features for the meta-model (level-1, final estimator). To keep the meta-model from overfitting on a leak, it trains on out-of-fold predictions: the data is split into k folds, each first-level model trains on k-1 parts and predicts the remaining one, and for production it is retrained on all the data. This is how StackingClassifier and StackingRegressor work in scikit-learn: they use cross_val_predict, and in a binary task they drop one of the two probability columns as collinear. The meta-model is often a regularized linear model, but boosting also works, and stacks can be nested. A variant where model weights depend on meta-features is called feature-weighted linear stacking (Sill et al., 2009). A stack is often at least as good as the strongest model in it, but training costs much more.
Example
A bank combines three borrower scoring models: one on the application form, one on payment history and one on behavior in the mobile app. The meta-model gets their three scores and learns that the form matters more for new clients, and payment history for long-standing ones.
Related terms
Questions and answers
How is stacking different from blending?
Sources
Not sure whether your company needs this?
AI solutions map