Glossary
What is the learning rate?
In short
The learning rate is a number that sets how strongly a model changes its weights at each training step.
Machine learning · learning rate
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine you are learning to throw a ball into a basket. You miss to the left, so next time you turn your hand a little to the right. If you turn it only a tiny bit, you will take a long time to score. If you jerk your hand hard, you will miss on the right, then on the left. You need to correct yourself just enough. A computer is also told how big its corrections should be.
In plain words
After each batch of examples, the model learns in which direction it must change to make fewer errors. But the direction does not say how far to step. A separate number decides that: the learning rate. A small value gives careful steps, so training is reliable but slow. A large value speeds it up, but the model can jump past the right point and never learn. So teams often start with large steps and make them smaller over time. A driver parks the same way: far from the wall they drive boldly, and near it they move centimeter by centimeter.
For business
This number decides how much fine-tuning costs and whether it works at all. A bad value wastes hours of GPU time: the model either barely learns or breaks and outputs garbage, and the run has to be repeated. When you fine-tune a ready model on your data, steps that are too sharp erase what it knew before, and the assistant starts to answer general questions worse. An experienced team picks the value on a short trial, not on the full run, and shows a chart of the error. Ask your contractor for this chart and which values they compared. If the line jumps or the error grows, training is set up badly. A pilot is a good place to check this before you pay for the full scope.
For a specialist
The learning rate (η) is a scalar multiplier of the step in gradient methods: θ := θ - η·g, where g is an estimate of the gradient of the loss function. Too large an η leads to divergence, too small an η to slow convergence and getting stuck on a plateau. The value usually changes by a learning rate schedule: step decay, cosine decay, SGDR warm restarts (Loshchilov, Hutter, 2016), and linear warmup at the start. Smith (2015) proposed a cyclical learning rate that swings between two limits, and a way to find those limits by raising η linearly over several epochs. Adaptive optimizers (AdaGrad, RMSProp, Adam) scale the step for each parameter by the history of gradients, but their base η is still set by hand. When the batch size grows, η is often raised in proportion: Goyal and co-authors (2017) trained a network with a batch of 8192 by this linear rule. In fine-tuning, η is clearly smaller than in training from scratch.
Example
A team fine-tunes a model to answer questions about internal rules. With a large step, the error rises sharply after a hundred iterations, and the model starts to repeat the same words. The team makes the step several times smaller, adds a smooth warmup, and training runs evenly.
Related terms
Questions and answers
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map