MMOOZZGG.com

Glossary

What is backpropagation?

In short

Backpropagation is the algorithm that computes how much each weight of a neural network contributed to an error, moving through the network from the output to the input.

Machine learning · backpropagation

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine your team lost a relay race. The coach reviews the race from the end: the last runner lost two seconds, the one before lost one, and the first ran very well. Now each runner knows what to practice. A computer learns in a similar way: it reviews its miss from the last step to the first and decides which part of itself to fix, and by how much.

In plain words

A neural network has layers, and each layer holds thousands of numbers called weights. When the network gives a wrong answer, it is not clear which weights are to blame. This algorithm answers that question. First, the data moves forward, from the input to the answer. Then the miss is sent back: the last layer learns how much it affected the result and passes this to the layer before, and so on to the first. In the end, each weight has a hint about which way to move and how far. It is like tracing a defect in a factory: you go from the finished part back through the machines it passed.

For business

This algorithm trains neural networks, from document recognition to language models, and for a company it matters through hardware and money. The backward pass needs to store the intermediate results of the forward pass. So training takes much more GPU memory than running a finished model, and it costs more. Hence a practical rule: using a ready model is cheap, fine-tuning it costs more, and training from scratch costs the most. Before a project, ask which layers will be fine-tuned, on which GPUs and for how many hours. If prompts and document search solve the task without training, the budget is much smaller. A pilot shows whether you need training at all.

For a specialist

Backpropagation computes the gradient of the loss function with respect to all network parameters in one backward pass, applying the chain rule layer by layer. The forward pass computes the activations and stores the intermediate values it needs. The backward pass goes from the output to the input and, for each layer, gets the gradients for the weights and for the layer’s input. In modern frameworks it is a special case of reverse-mode automatic differentiation (reverse-mode autodiff). According to the PyTorch documentation, autograd records operations in a directed acyclic graph and traverses it from the roots to the leaves. Rumelhart, Hinton and Williams described the procedure for multilayer networks in Nature (9 October 1986). They showed that hidden neurons trained this way start to represent useful features of the task on their own. An optimizer, such as stochastic gradient descent or Adam, takes the step along the gradient. Known problems: vanishing and exploding gradients in deep networks, and the memory for stored activations, which recomputation (gradient checkpointing) saves.

Example

A network that read scanned invoices confused the digit one with seven. After each batch of examples, backpropagation showed which weights in which layers pulled hardest toward the miss, and the optimizer corrected them. After a few epochs, the confusion almost disappeared.

Related terms

Questions and answers

Why do we need backpropagation? Without it, you would have to test the effect of each weight separately, and a network has millions of them. The algorithm gets all the corrections at once in one backward pass.

That is why training large networks takes a reasonable time.

How is backpropagation different from gradient descent? Backpropagation only computes the gradient: the direction and size of the corrections. Gradient descent or another optimizer takes this gradient and makes a step.

In training, the two work as a pair.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us