Glossary
Ensemble learning: what a model ensemble is
In short
Ensemble learning means several machine learning models whose answers are combined into one forecast, so that it is more accurate and more stable than that of any single model.
Machine learning · ensemble
MMOOZZGG.com editorsChecked: 2 min read
For a child
When a class is asked how many candies are in a jar, one student can be far off. But if you write down everyone’s guesses and take the average, the answer is often surprisingly close. Someone said too many, someone too few, and the misses cancel each other out. Computers sometimes consult each other in the same way: several programs give an answer, and then their opinions are added into one.
In plain words
Every model makes errors, but different models make different errors. If you combine their answers, random misses partly cancel each other out. There are several ways to combine them. You can hold a vote: the option most models named is the answer. You can average numbers, for example sales forecasts from three different models. You can train models one after another, so that each next one fixes the previous one. Or you can put one more model on top as a referee that decides whom to trust and how much. A panel of doctors works this way: the group’s decision is more reliable than one specialist’s opinion if the doctors look at the case from different sides.
For business
Ensembles add accuracy where every fraction of a percent costs money: demand and purchase forecasts, credit scoring, fraud detection, product ranking. You pay for the gain with complexity. Several models take longer to train, cost more to run and are harder to explain to an auditor or a client. If the difference from one good model is small, it is simpler to keep one. Before rollout, compare the combined system with a simple baseline on fresh data and calculate whether the gain pays for the upkeep. Such a comparison for demand forecasting and other predictive tasks is planned already at the pilot stage.
For a specialist
An ensemble combines the predictions of several base estimators to improve generalization and stability compared with a single model. The gain appears when the members’ errors are weakly correlated. Averaging methods reduce variance: bagging trains models on bootstrap samples and was proposed by L. Breiman (1996); a random forest adds random feature selection at each split; voting can be hard, by majority of labels, or soft, by averaged probabilities. Bagging helps more with complex models, boosting with weak ones. Boosting trains weak models one after another and reduces bias: AdaBoost, gradient boosting. Stacking trains a meta-model on the members’ out-of-sample predictions, and blending does the same on a held-out part of the data. A key part of the solution that took second place in the Netflix Prize was linear stacking with weights that depend on meta-features (Sill et al., 2009).
Example
A retail chain forecasts sales with three models: one catches seasonality, the second reacts to promotions, and the third accounts for the weather. The final forecast is a weighted average, and on holidays it misses less often than any of the three models alone.
Related terms
Questions and answers
Is a combination of models always more accurate than one model?
Sources
Not sure whether your company needs this?
AI solutions map