MMOOZZGG.com

Glossary

What is a gradient in machine learning?

In short

A gradient is a set of numbers that shows in which direction and how fast a model’s error changes when each of its parameters changes a little.

Math of AI · gradient

MMOOZZGG.com editorsChecked: 1 min read

For a child

You are sledding and want to find the place where the hill is steeper. If you stand at the top and look around, you can see where the slope goes down more sharply. Such a hint, it is steeper here, helps you choose where to go. When a computer learns, it looks for a similar hint, only its slope is invisible: it is made of its mistakes, and it wants to get lower.

In plain words

Imagine a control panel with a million knobs, and how accurately the model answers depends on the position of each. Turning them at random makes no sense. This hint tells two things about each knob: which way a turn will increase the error, and how strongly the knob affects it. The model turns all the knobs a little in the opposite direction, and the error goes down. Then the hint is recalculated on new examples. Doing this by hand is impossible, so a program calculates it automatically, going through the network from the answer back to the input.

For business

Calculating these corrections is the main part of the cost of training and fine-tuning models. At each step, the GPU runs the data forward, then calculates corrections for all weights, and for that it must keep intermediate results in memory. So training needs noticeably more video memory than running a ready model, and therefore a more expensive server. This matters when you choose between fine-tuning your own model and setting up a ready one through instructions and document search. Methods such as LoRA freeze the original weights and train only small added matrices, so hardware requirements are lower. A configuration for training is chosen separately from a configuration for answers.

For a specialist

The gradient of a differentiable function of many variables is the vector of its partial derivatives with respect to each variable. It points in the direction of the fastest growth of the function, so optimization methods move parameters against it. In neural networks the function is the loss, and the variables are millions or billions of weights. All partial derivatives in one backward pass come from backpropagation, described by Rumelhart, Hinton and Williams in Nature in 1986. Today it is computed by reverse-mode automatic differentiation; a survey by Baydin et al. (JMLR, 2018) describes this family of methods as more general than backpropagation. In deep and recurrent networks, gradients can vanish (vanishing gradients) or explode (exploding gradients); Pascanu, Mikolov and Bengio (2012) analyzed both problems and proposed norm clipping (gradient clipping) against explosion. In practice, a stochastic estimate on a mini-batch is used, and adaptive optimizers scale it by the history of past steps.

Do not confuse. Not to be confused with a gradient in design, a smooth change from one color to another.

Example

While fine-tuning a model on support answers, an engineer notices that the weight corrections have jumped and the error swings. He turns on gradient clipping and lowers the learning rate, after which training runs smoothly again.

Related terms

Questions and answers

What is a gradient in machine learning? It is a set of numbers, one for each model parameter, that shows how the error will change if you move that parameter a little. From it, the model learns how to adjust its weights.
How is a gradient different from gradient descent? The first is a direction, a hint. Descent is a training method that again and again takes a step against that direction.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us