Glossary
What is gradient boosting
In short
Gradient boosting is a machine learning method that builds many small decision trees one after another, and each new tree fixes the errors of the trees already built.
Machine learning · gradient boosting
MMOOZZGG.com editorsChecked: 2 min read
For a child
A class draws a big picture in turns. The first pupil sketches the outline, and it comes out crooked. The second does not start over but fixes only what the first got wrong. The third fixes the second one’s misses. Each takes a small step, but after twenty pupils the picture is accurate. A computer learns in a similar way: many small helpers, and each fixes someone else’s mistakes.
In plain words
One simple decision tree asks a few questions and gives an answer, for example whether a client will buy or not. It is often wrong. This method takes such a tree, looks at where the forecast differed from reality and trains the next tree on exactly those misses. Then a third, a fourth, and so on hundreds of times. The final answer adds up the contributions of all the trees, and the contribution of each is made smaller on purpose so that the model learns carefully. It is like adjusting a recipe: you try it, notice it is too salty and add a little less salt next time.
For business
For tables from accounting systems (1C in Russia) and the CRM, this approach often gives an accurate forecast at modest cost: the model trains on an ordinary server without graphics cards in minutes or hours. It is used for client scoring, churn and demand forecasts, detecting suspicious transactions and estimating the chance of a deal. The project price depends mostly on data preparation: how many sources there are, whether there is a history with correct answers and how clean it is. Before training, an exploratory data analysis finds gaps and fields that accidentally hint at the answer. The main risk: the model looks great on the past and is wrong on new months, so it is tested on a period it has not seen. Such scenarios, with the effect estimated in money, are covered in the AI roadmap.
For a specialist
Gradient boosting builds an additive model from weak learners, usually shallow decision trees. At each step, a new tree is trained to predict the negative gradient of the loss function at the current predictions, so the method is described as descent in function space; for squared error, the negative gradient equals the residuals. The tree’s contribution is multiplied by the learning rate (shrinkage): a smaller rate needs more trees but usually lowers the test error. The stochastic variant trains each tree on a random fraction of the sample without replacement. Friedman (J. H. Friedman, 2001) described the general scheme for any differentiable loss function. Industrial implementations speed up the search for splits: XGBoost (Chen, Guestrin, 2016) added an algorithm for sparse data and a weighted quantile sketch, and LightGBM (Ke et al., 2017) introduced gradient-based one-side sampling (GOSS) and exclusive feature bundling (EFB). Unlike linear models, trees do not need data normalization. Overfitting is held back by early stopping, depth limits and the subsample fraction.
Example
An online store trains a model on its order history: purchase frequency, average order value, days since the last visit. The model ranks clients by their chance of leaving, and the marketer sends a discount only to the top of the list, not to the whole base.
Related terms
Questions and answers
How is gradient boosting different from a random forest?
Sources
- scikit-learn, User Guide: Ensembles (Gradient-boosted trees, citing Friedman, 2001), checked 16 October 2026scikit-learn.org
- Chen, Guestrin, XGBoost: A Scalable Tree Boosting System, 2016arxiv.org
- Ke et al., LightGBM: A Highly Efficient Gradient Boosting Decision Tree, NeurIPS 2017proceedings.neurips.cc
Not sure whether your company needs this?
AI solutions map