Glossary
What is the gradient descent algorithm
In short
The gradient descent algorithm is a way of training a model in which its internal numbers are shifted step by step toward where the error falls fastest.
Machine learning · gradient descent
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine you are walking down a mountain in thick fog and cannot see the bottom. You feel the ground with your foot, find where the slope goes down most steeply and take a small step that way. Then you feel again and step again. Step by step, you reach the valley. A computer learns in a similar way, only instead of a mountain it has errors.
In plain words
A model learns from examples and at first often makes mistakes. The error can be expressed as one number: the larger it is, the worse the answers. Gradient descent shows how to change the model’s settings so that this number gets smaller. The computer looks at which way the error grows and moves each setting a little the other way. Then it takes new examples and repeats. After thousands of such small corrections, the model answers much more accurately. It is like tuning an old radio: you turn the knob toward where the noise is quieter until you catch a clear sound.
For business
Almost all neural networks that companies buy or order are trained with this method, so the compute bill and the project timeline depend on it. Each step needs calculations on graphics cards, and thousands or millions of steps are needed, so the training cost grows with the amount of data and the size of the model. For a business this matters in two cases: when you fine-tune a model on your own data and when you assess a contractor’s estimate. Ask how many GPU hours are planned, how the team will know when to stop training and on which data the result will be checked. If the error on the validation data has stopped falling, further steps only spend the budget. Such calculations are easy to do in a pilot, before you buy servers.
For a specialist
Gradient descent is an iterative method for minimizing a differentiable loss function: the parameters are updated by the rule θ := θ - η·∇L(θ), where η is the learning rate. The full variant (batch gradient descent) computes the gradient over the whole sample, the stochastic one (SGD) over one example, and the mini-batch one over a small group of examples; deep learning almost always uses the last. The gradient with respect to the network weights is obtained by backpropagation. The plain method moves slowly along narrow ravines of the loss surface, so modifications are used: momentum, Nesterov accelerated gradient, and the adaptive methods RMSProp and Adam, which Kingma and Ba proposed in 2014. Sebastian Ruder gave an overview of descent variants and popular optimizers in 2016. For the non-convex functions of neural networks, the method does not necessarily find the global minimum and converges to one of the local solutions, and the result depends on initialization, batch size and the step schedule.
Example
A bank trains a model to assess the risk of default from borrowers’ history. In the first steps of gradient descent, the model mixes up reliable and risky clients almost at random, but after several passes over the data the error on the validation sample falls and stops changing, and training is stopped.
Related terms
Questions and answers
How is stochastic gradient descent different from the ordinary kind?
Sources
Not sure whether your company needs this?
AI solutions map