MMOOZZGG.com

Glossary

What is the learning rate?

In short

The learning rate is a number that sets how strongly a model changes its weights at each training step.

Machine learning · learning rate

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you are learning to throw a ball into a basket. You miss to the left, so next time you turn your hand a little to the right. If you turn it only a tiny bit, you will take a long time to score. If you jerk your hand hard, you will miss on the right, then on the left. You need to correct yourself just enough. A computer is also told how big its corrections should be.

In plain words

After each batch of examples, the model learns in which direction it must change to make fewer errors. But the direction does not say how far to step. A separate number decides that: the learning rate. A small value gives careful steps, so training is reliable but slow. A large value speeds it up, but the model can jump past the right point and never learn. So teams often start with large steps and make them smaller over time. A driver parks the same way: far from the wall they drive boldly, and near it they move centimeter by centimeter.

For business

This number decides how much fine-tuning costs and whether it works at all. A bad value wastes hours of GPU time: the model either barely learns or breaks and outputs garbage, and the run has to be repeated. When you fine-tune a ready model on your data, steps that are too sharp erase what it knew before, and the assistant starts to answer general questions worse. An experienced team picks the value on a short trial, not on the full run, and shows a chart of the error. Ask your contractor for this chart and which values they compared. If the line jumps or the error grows, training is set up badly. A pilot is a good place to check this before you pay for the full scope.

For a specialist

The learning rate (η) is a scalar multiplier of the step in gradient methods: θ := θ - η·g, where g is an estimate of the gradient of the loss function. Too large an η leads to divergence, too small an η to slow convergence and getting stuck on a plateau. The value usually changes by a learning rate schedule: step decay, cosine decay, SGDR warm restarts (Loshchilov, Hutter, 2016), and linear warmup at the start. Smith (2015) proposed a cyclical learning rate that swings between two limits, and a way to find those limits by raising η linearly over several epochs. Adaptive optimizers (AdaGrad, RMSProp, Adam) scale the step for each parameter by the history of gradients, but their base η is still set by hand. When the batch size grows, η is often raised in proportion: Goyal and co-authors (2017) trained a network with a batch of 8192 by this linear rule. In fine-tuning, η is clearly smaller than in training from scratch.

Example

A team fine-tunes a model to answer questions about internal rules. With a large step, the error rises sharply after a hundred iterations, and the model starts to repeat the same words. The team makes the step several times smaller, adds a smooth warmup, and training runs evenly.

Related terms

Questions and answers

Which learning rate should you choose? There is no single value: it depends on the model, the optimizer and the batch size. Usually you take the value from the model documentation, or find it with a short trial: raise the step slowly and watch where the error falls fastest.
What happens if the learning rate is too high? The model jumps over good solutions, the error jumps or grows, and in bad cases the weights turn into meaningless numbers. You must stop training and restart it with a smaller step.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us