Glossary
What are scaling laws in AI?
In short
Scaling laws in AI are observed relations by which the quality of a language model rises in a predictable way as the number of parameters, the amount of data and the compute grow.
Language models · scaling laws
MMOOZZGG.com editorsChecked: 1 min read
For a child
When you build a tower of blocks, the more blocks and time you have, the higher it gets. Scientists noticed something similar with neural networks: if you give them more examples to read and more time to learn, they make fewer mistakes. And they improve not at random but by a clear rule, which you can draw on a chart in advance.
In plain words
Researchers trained many models of different sizes and saw a pattern. The error falls smoothly and predictably as three things grow: the number of parameters, the amount of training text and the computing power. On a special chart this looks almost like a straight line. So small test models let you estimate what a large one will achieve before its expensive training. But each next step needs many times more resources, and the gain from it slowly shrinks.
For business
These relations explain why frontier models cost so much: to cut the error noticeably, a developer needs many times more graphics cards, electricity and data. For a company that uses models, the lesson is practical. You do not always need a huge model: on a narrow task, a smaller model trained on more data gives similar quality and costs less to run. The price of each answer depends on the model size, so the size you pick affects the budget every month. The risk is that the charts describe the average error, not your task. So you choose the model by measuring it on your own examples, and then pick a server for it.
For a specialist
Scaling laws link the loss function to the number of parameters N, the amount of data D and the compute budget C. Kaplan et al. (2020) showed that cross-entropy falls as a power law in each of the three factors, with some trends traced over more than seven orders of magnitude, while the width and depth of the network matter little over a wide range. Their estimates suggested it was better to train very large models on a moderate amount of data. Hoffmann et al. (2022) revised this: after training more than 400 models, they found that at the optimal budget, parameters and tokens should grow in equal proportion. The test was Chinchilla with 70 billion parameters: with the same compute as the 280-billion-parameter Gopher and four times more data, it beat Gopher on most tasks. The law predicts the average loss but not individual skills: jumps in quality are debated under the topic of emergent abilities.
Example
A team chooses a model to sort customer requests. Instead of the largest available model, it tests three sizes on a thousand of its own emails and sees that the medium model makes almost as few mistakes, while each answer costs several times less.
Related terms
Questions and answers
Do scaling laws work forever?
Sources
Not sure whether your company needs this?
AI solutions map