MMOOZZGG.com

Glossary

What is the gradient descent algorithm

In short

The gradient descent algorithm is a way of training a model in which its internal numbers are shifted step by step toward where the error falls fastest.

Machine learning · gradient descent

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine you are walking down a mountain in thick fog and cannot see the bottom. You feel the ground with your foot, find where the slope goes down most steeply and take a small step that way. Then you feel again and step again. Step by step, you reach the valley. A computer learns in a similar way, only instead of a mountain it has errors.

In plain words

A model learns from examples and at first often makes mistakes. The error can be expressed as one number: the larger it is, the worse the answers. Gradient descent shows how to change the model’s settings so that this number gets smaller. The computer looks at which way the error grows and moves each setting a little the other way. Then it takes new examples and repeats. After thousands of such small corrections, the model answers much more accurately. It is like tuning an old radio: you turn the knob toward where the noise is quieter until you catch a clear sound.

For business

Almost all neural networks that companies buy or order are trained with this method, so the compute bill and the project timeline depend on it. Each step needs calculations on graphics cards, and thousands or millions of steps are needed, so the training cost grows with the amount of data and the size of the model. For a business this matters in two cases: when you fine-tune a model on your own data and when you assess a contractor’s estimate. Ask how many GPU hours are planned, how the team will know when to stop training and on which data the result will be checked. If the error on the validation data has stopped falling, further steps only spend the budget. Such calculations are easy to do in a pilot, before you buy servers.

For a specialist

Gradient descent is an iterative method for minimizing a differentiable loss function: the parameters are updated by the rule θ := θ - η·∇L(θ), where η is the learning rate. The full variant (batch gradient descent) computes the gradient over the whole sample, the stochastic one (SGD) over one example, and the mini-batch one over a small group of examples; deep learning almost always uses the last. The gradient with respect to the network weights is obtained by backpropagation. The plain method moves slowly along narrow ravines of the loss surface, so modifications are used: momentum, Nesterov accelerated gradient, and the adaptive methods RMSProp and Adam, which Kingma and Ba proposed in 2014. Sebastian Ruder gave an overview of descent variants and popular optimizers in 2016. For the non-convex functions of neural networks, the method does not necessarily find the global minimum and converges to one of the local solutions, and the result depends on initialization, batch size and the step schedule.

Example

A bank trains a model to assess the risk of default from borrowers’ history. In the first steps of gradient descent, the model mixes up reliable and risky clients almost at random, but after several passes over the data the error on the validation sample falls and stops changing, and training is stopped.

Related terms

Questions and answers

Why is gradient descent needed? It tunes millions of numbers inside a model so that it makes mistakes less often. Trying such numbers by hand is impossible, and the gradient shows right away the direction in which the error goes down.
How is stochastic gradient descent different from the ordinary kind? The ordinary kind computes the direction from all training examples at once: this is precise but slow. The stochastic kind takes one example or a small group, so the steps are noisy but fast.

On large data, this variant reaches a good model sooner.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us