MMOOZZGG.com

Glossary

What is LLM quantization

In short

LLM quantization is a way to compress a neural network by storing its numbers with lower precision, so the model takes less memory and runs faster.

Language models · quantization

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you are drawing a rainbow. A big box has a hundred pencils of almost the same shades, and a small box has only ten. A drawing made with the small box comes out a little simpler, but the box fits easily into your backpack. A neural network can also move into a small box: it becomes lighter and faster, and its answers barely change.

In plain words

A language model holds billions of numbers, and each is written in great detail, with many digits after the decimal point. Quantization rounds them: instead of a long record, a short one is kept. The model gets two to four times smaller and fits on an ordinary graphics card or even a laptop. The meaning of answers suffers little, because the small tails of the numbers do not matter, only their general order. It is like a photo in a compressed format: the file is lighter, and the face is still recognizable.

For business

For a company this is a question of money for hardware. A model that needs a server with several expensive graphics cards at full precision often runs on one card after compression to 8 or 4 bits. This makes it cheaper to run AI on your own servers, where customer data does not leave. The price depends on the model size, the chosen number of bits and the load: how many employees ask at the same time. The risk is that strong compression damages rare skills: long calculations, exact quotes, answers on narrow topics. So compare the compressed version with the original on your real questions before you buy equipment. It helps to choose the graphics card for a specific model in advance, by a memory calculation.

For a specialist

Quantization moves the weights, and sometimes the activations, of a model from a floating-point format (FP16, BF16) to low-precision integers (INT8, INT4), with a scale and a zero point per group of values. There are two kinds: post-training quantization (PTQ) and quantization-aware training (QAT). For large language models the main difficulty is outliers: rare features with a large amplitude break the shared scale. LLM.int8() (Dettmers et al., 2022) keeps such dimensions in 16 bits, multiplies the rest in 8 bits, and halves inference memory with no loss of quality. GPTQ (Frantar et al., 2022) uses approximate second-order information and compresses the weights of a 175-billion-parameter model to 3–4 bits in about four GPU hours. Related approaches: AWQ, the GGUF format for local launch, and QLoRA for fine-tuning on top of a 4-bit model.

Example

A company needs an assistant on internal rules on its own server. The model at full precision does not fit into the memory of one graphics card, but the 4-bit version fits and answers test questions almost as well. The company buys one card instead of two.

Related terms

Questions and answers

How is quantization different from distillation? Quantization keeps the same model and only rounds its numbers. Distillation trains a new, smaller model to repeat the answers of a large one.

The two methods are often used together.

How much does quality drop after quantization? At 8 bits the loss is usually almost invisible. At 3–4 bits it is small if the method is good, and below that quality drops more clearly.

Test it on your own tasks, not on general tables.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us