MMOOZZGG.com

Glossary

What are scaling laws in AI?

In short

Scaling laws in AI are observed relations by which the quality of a language model rises in a predictable way as the number of parameters, the amount of data and the compute grow.

Language models · scaling laws

MMOOZZGG.com editorsChecked: 1 min read

For a child

When you build a tower of blocks, the more blocks and time you have, the higher it gets. Scientists noticed something similar with neural networks: if you give them more examples to read and more time to learn, they make fewer mistakes. And they improve not at random but by a clear rule, which you can draw on a chart in advance.

In plain words

Researchers trained many models of different sizes and saw a pattern. The error falls smoothly and predictably as three things grow: the number of parameters, the amount of training text and the computing power. On a special chart this looks almost like a straight line. So small test models let you estimate what a large one will achieve before its expensive training. But each next step needs many times more resources, and the gain from it slowly shrinks.

For business

These relations explain why frontier models cost so much: to cut the error noticeably, a developer needs many times more graphics cards, electricity and data. For a company that uses models, the lesson is practical. You do not always need a huge model: on a narrow task, a smaller model trained on more data gives similar quality and costs less to run. The price of each answer depends on the model size, so the size you pick affects the budget every month. The risk is that the charts describe the average error, not your task. So you choose the model by measuring it on your own examples, and then pick a server for it.

For a specialist

Scaling laws link the loss function to the number of parameters N, the amount of data D and the compute budget C. Kaplan et al. (2020) showed that cross-entropy falls as a power law in each of the three factors, with some trends traced over more than seven orders of magnitude, while the width and depth of the network matter little over a wide range. Their estimates suggested it was better to train very large models on a moderate amount of data. Hoffmann et al. (2022) revised this: after training more than 400 models, they found that at the optimal budget, parameters and tokens should grow in equal proportion. The test was Chinchilla with 70 billion parameters: with the same compute as the 280-billion-parameter Gopher and four times more data, it beat Gopher on most tasks. The law predicts the average loss but not individual skills: jumps in quality are debated under the topic of emergent abilities.

Example

A team chooses a model to sort customer requests. Instead of the largest available model, it tests three sizes on a thousand of its own emails and sees that the medium model makes almost as few mistakes, while each answer costs several times less.

Related terms

Questions and answers

Do scaling laws work forever? Nobody knows the limit yet. A power-law relation means that each further cut in error needs many times more resources.

Also, the amount of good-quality text on the internet is finite, and developers look for new data sources, including synthetic data.

What is the Chinchilla law? This is the name for the finding of a 2022 paper: with a fixed compute budget, you should grow the number of parameters and the amount of data equally. The authors showed that many large models of that time were heavily undertrained.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us