MMOOZZGG.com

Glossary

Ensemble learning: what a model ensemble is

In short

Ensemble learning means several machine learning models whose answers are combined into one forecast, so that it is more accurate and more stable than that of any single model.

Machine learning · ensemble

MMOOZZGG.com editorsChecked: 2 min read

For a child

When a class is asked how many candies are in a jar, one student can be far off. But if you write down everyone’s guesses and take the average, the answer is often surprisingly close. Someone said too many, someone too few, and the misses cancel each other out. Computers sometimes consult each other in the same way: several programs give an answer, and then their opinions are added into one.

In plain words

Every model makes errors, but different models make different errors. If you combine their answers, random misses partly cancel each other out. There are several ways to combine them. You can hold a vote: the option most models named is the answer. You can average numbers, for example sales forecasts from three different models. You can train models one after another, so that each next one fixes the previous one. Or you can put one more model on top as a referee that decides whom to trust and how much. A panel of doctors works this way: the group’s decision is more reliable than one specialist’s opinion if the doctors look at the case from different sides.

For business

Ensembles add accuracy where every fraction of a percent costs money: demand and purchase forecasts, credit scoring, fraud detection, product ranking. You pay for the gain with complexity. Several models take longer to train, cost more to run and are harder to explain to an auditor or a client. If the difference from one good model is small, it is simpler to keep one. Before rollout, compare the combined system with a simple baseline on fresh data and calculate whether the gain pays for the upkeep. Such a comparison for demand forecasting and other predictive tasks is planned already at the pilot stage.

For a specialist

An ensemble combines the predictions of several base estimators to improve generalization and stability compared with a single model. The gain appears when the members’ errors are weakly correlated. Averaging methods reduce variance: bagging trains models on bootstrap samples and was proposed by L. Breiman (1996); a random forest adds random feature selection at each split; voting can be hard, by majority of labels, or soft, by averaged probabilities. Bagging helps more with complex models, boosting with weak ones. Boosting trains weak models one after another and reduces bias: AdaBoost, gradient boosting. Stacking trains a meta-model on the members’ out-of-sample predictions, and blending does the same on a held-out part of the data. A key part of the solution that took second place in the Netflix Prize was linear stacking with weights that depend on meta-features (Sill et al., 2009).

Example

A retail chain forecasts sales with three models: one catches seasonality, the second reacts to promotions, and the third accounts for the weather. The final forecast is a weighted average, and on holidays it misses less often than any of the three models alone.

Related terms

Questions and answers

What types of model ensembles exist? The main types are bagging and random forests, where models learn in parallel and are averaged; boosting, where they learn one after another; and voting and stacking. The choice depends on what you need to reduce: the spread of forecasts or the systematic error.
Is a combination of models always more accurate than one model? No. If the models are similar and make errors on the same examples, combining them gives almost nothing.

The gain appears when the members differ, and its size is checked on held-out data.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us