Glossary
What is backpropagation?
In short
Backpropagation is the algorithm that computes how much each weight of a neural network contributed to an error, moving through the network from the output to the input.
Machine learning · backpropagation
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine your team lost a relay race. The coach reviews the race from the end: the last runner lost two seconds, the one before lost one, and the first ran very well. Now each runner knows what to practice. A computer learns in a similar way: it reviews its miss from the last step to the first and decides which part of itself to fix, and by how much.
In plain words
A neural network has layers, and each layer holds thousands of numbers called weights. When the network gives a wrong answer, it is not clear which weights are to blame. This algorithm answers that question. First, the data moves forward, from the input to the answer. Then the miss is sent back: the last layer learns how much it affected the result and passes this to the layer before, and so on to the first. In the end, each weight has a hint about which way to move and how far. It is like tracing a defect in a factory: you go from the finished part back through the machines it passed.
For business
This algorithm trains neural networks, from document recognition to language models, and for a company it matters through hardware and money. The backward pass needs to store the intermediate results of the forward pass. So training takes much more GPU memory than running a finished model, and it costs more. Hence a practical rule: using a ready model is cheap, fine-tuning it costs more, and training from scratch costs the most. Before a project, ask which layers will be fine-tuned, on which GPUs and for how many hours. If prompts and document search solve the task without training, the budget is much smaller. A pilot shows whether you need training at all.
For a specialist
Backpropagation computes the gradient of the loss function with respect to all network parameters in one backward pass, applying the chain rule layer by layer. The forward pass computes the activations and stores the intermediate values it needs. The backward pass goes from the output to the input and, for each layer, gets the gradients for the weights and for the layer’s input. In modern frameworks it is a special case of reverse-mode automatic differentiation (reverse-mode autodiff). According to the PyTorch documentation, autograd records operations in a directed acyclic graph and traverses it from the roots to the leaves. Rumelhart, Hinton and Williams described the procedure for multilayer networks in Nature (9 October 1986). They showed that hidden neurons trained this way start to represent useful features of the task on their own. An optimizer, such as stochastic gradient descent or Adam, takes the step along the gradient. Known problems: vanishing and exploding gradients in deep networks, and the memory for stored activations, which recomputation (gradient checkpointing) saves.
Example
A network that read scanned invoices confused the digit one with seven. After each batch of examples, backpropagation showed which weights in which layers pulled hardest toward the miss, and the optimizer corrected them. After a few epochs, the confusion almost disappeared.
Related terms
Questions and answers
Why do we need backpropagation?
How is backpropagation different from gradient descent?
Sources
Not sure whether your company needs this?
AI solutions map