MMOOZZGG.com

Glossary

What is gradient boosting

In short

Gradient boosting is a machine learning method that builds many small decision trees one after another, and each new tree fixes the errors of the trees already built.

Machine learning · gradient boosting

MMOOZZGG.com editorsChecked: 2 min read

For a child

A class draws a big picture in turns. The first pupil sketches the outline, and it comes out crooked. The second does not start over but fixes only what the first got wrong. The third fixes the second one’s misses. Each takes a small step, but after twenty pupils the picture is accurate. A computer learns in a similar way: many small helpers, and each fixes someone else’s mistakes.

In plain words

One simple decision tree asks a few questions and gives an answer, for example whether a client will buy or not. It is often wrong. This method takes such a tree, looks at where the forecast differed from reality and trains the next tree on exactly those misses. Then a third, a fourth, and so on hundreds of times. The final answer adds up the contributions of all the trees, and the contribution of each is made smaller on purpose so that the model learns carefully. It is like adjusting a recipe: you try it, notice it is too salty and add a little less salt next time.

For business

For tables from accounting systems (1C in Russia) and the CRM, this approach often gives an accurate forecast at modest cost: the model trains on an ordinary server without graphics cards in minutes or hours. It is used for client scoring, churn and demand forecasts, detecting suspicious transactions and estimating the chance of a deal. The project price depends mostly on data preparation: how many sources there are, whether there is a history with correct answers and how clean it is. Before training, an exploratory data analysis finds gaps and fields that accidentally hint at the answer. The main risk: the model looks great on the past and is wrong on new months, so it is tested on a period it has not seen. Such scenarios, with the effect estimated in money, are covered in the AI roadmap.

For a specialist

Gradient boosting builds an additive model from weak learners, usually shallow decision trees. At each step, a new tree is trained to predict the negative gradient of the loss function at the current predictions, so the method is described as descent in function space; for squared error, the negative gradient equals the residuals. The tree’s contribution is multiplied by the learning rate (shrinkage): a smaller rate needs more trees but usually lowers the test error. The stochastic variant trains each tree on a random fraction of the sample without replacement. Friedman (J. H. Friedman, 2001) described the general scheme for any differentiable loss function. Industrial implementations speed up the search for splits: XGBoost (Chen, Guestrin, 2016) added an algorithm for sparse data and a weighted quantile sketch, and LightGBM (Ke et al., 2017) introduced gradient-based one-side sampling (GOSS) and exclusive feature bundling (EFB). Unlike linear models, trees do not need data normalization. Overfitting is held back by early stopping, depth limits and the subsample fraction.

Example

An online store trains a model on its order history: purchase frequency, average order value, days since the last visit. The model ranks clients by their chance of leaving, and the marketer sends a discount only to the top of the list, not to the whole base.

Related terms

Questions and answers

How is gradient boosting different from a random forest? A random forest builds trees independently and averages their answers, while boosting builds them one after another, and each next tree fixes the errors of the previous ones. A forest is easier to tune and overfits less often.

After careful tuning, boosting often gives a more accurate forecast on tabular data.

Does boosting need graphics cards? No, it is a classic machine learning method, and it runs on ordinary processors. Graphics cards speed up training on very large tables, but for most company tasks an ordinary server is enough.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us