Glossary
What are model parameters?
In short
Model parameters are the numbers inside a neural network that it sets during training and then uses to compute every answer.
Language models · parameters
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a huge mixing desk with thousands of small knobs, like a sound engineer has at a concert. At first all the knobs are set at random, and the music sounds awful. The engineer turns each knob for a long time until the song sounds beautiful. A neural network has knobs like these too, only instead of volume they set which words go next to each other.
In plain words
A neural network holds billions of numbers, called parameters or weights. During training, the program compares its answer with the right one again and again. It moves these numbers a tiny bit so that it makes a smaller mistake next time. When training ends, the numbers are frozen and saved to a file. Each of your requests passes through them as through a huge formula. The more numbers there are, the more patterns the model can catch, but the file is also heavier: a model with tens of billions of numbers will not run on an ordinary laptop.
For business
The parameter count tells you what hardware your own model will need. Each parameter takes graphics card memory: at 16-bit precision that is two bytes, so a 70-billion-parameter model needs about 140 GB for the weights alone, with no headroom for requests. Quantization compresses the weights two to four times at the cost of a small loss in quality. Size does not equal value: Hoffmann and co-authors (2022) showed that Chinchilla, with 70 billion parameters, beat Gopher with 280 billion, because it trained on more data. So choose the model by its results on your tasks, and size the server for the chosen model before you buy hardware.
For a specialist
Parameters are the trainable values of a model: the weights and biases of linear layers, embedding matrices and normalization coefficients. In a transformer, most of them sit in the attention projections and the feed-forward blocks, and their number grows in proportion to the square of the layer width and to the number of layers. Their values are set by gradient descent on a loss function, while an engineer sets the hyperparameters, such as the learning rate and batch size. Kaplan and co-authors (2020) found a power-law dependence of the error on the number of parameters, data and compute. Based on training more than 400 models, Hoffmann and co-authors (2022) recommended doubling the training tokens with each doubling of model size. In a mixture of experts, the total parameter count differs from the active parameters per token. Memory for the weights equals the parameter count times the bytes per parameter: 2 for FP16 and BF16, about 0.5 for 4-bit quantization.
Example
A company was choosing a model for its own server. The 70-billion-parameter option at 16-bit precision needed two 80 GB graphics cards. A model half that size answered almost as well in a test on a hundred client emails and fit on one card.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map