MMOOZZGG.com

Glossary

What are emergent abilities in LLMs

In short

Emergent abilities in LLMs are skills that small language models lack and large ones show, and that could not be predicted from the results of smaller models.

Language models · emergent abilities

MMOOZZGG.com editorsChecked: 1 min read

For a child

You learn to ride a bike for a long time, you fall, and it seems nothing works. Then one day you ride by yourself, as if the skill came all at once. Scientists saw something similar with neural networks: a small one cannot solve a task at all, and a big one suddenly can. But some think the skill grew little by little, and it was just hard to see.

In plain words

When models get bigger, they usually become a little more accurate step by step. But on some tasks the picture is different: smaller models answer almost at random, and from a certain size they suddenly start to succeed. This applies, for example, to tasks with several steps. Such skills were called emergent, meaning “arising”. Later some scientists showed that the jump is often created by the way of scoring: if you give a point only for a fully correct answer, smooth growth looks sudden.

For business

For business, this debate teaches caution with forecasts. You cannot reliably predict whether the next version of a model will learn your task, and you cannot assume that a skill shown in a demo stays stable. The other side matters too: a small model may fail a task completely, not just do slightly worse. So choose the model size by testing on your real cases, with a quality threshold agreed in advance. Such a test costs little compared with the price of an error in real work. After a model change, repeat the tests, because skills can both appear and disappear.

For a specialist

Wei and coauthors (2022) defined emergent abilities as abilities that are absent in smaller models and present in larger ones, so they cannot be predicted by extrapolating the quality of small models. On a chart of accuracy against training compute such tasks look like a sharp jump from random level to a clearly higher one. The authors also included prompting methods, such as step-by-step reasoning (chain-of-thought), which helps only from a certain scale. Schaeffer, Miranda and Koyejo (2023) objected that the effect is often created by the researcher’s choice of metric. Discontinuous or nonlinear metrics such as exact match give a step, while continuous ones show smooth and predictable growth. They tested this on InstructGPT and GPT-3, a meta-analysis of BIG-Bench and vision tasks. The lesson for evaluation: look at continuous metrics and answer probabilities, not only at the share of fully correct answers.

Example

A company tested two models of different sizes on handling complex claims. The smaller one almost always mixed up the order of steps, while the larger one coped, so the company kept the larger one for this process and the smaller one for simple emails.

Related terms

Questions and answers

Do emergent abilities really exist? The question is disputed. Jumps in quality were seen on many tasks, but some researchers showed that with smooth metrics the growth looks gradual.

For practice something else matters: test the model’s skill on your own task.

Why do emergent abilities matter for business? They remind you that a small model may be unable to do a task at all, not just do it slightly worse. So choose the model size by tests on real examples, not by public rankings.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us