The formula
Weights: the number of parameters times the bytes per parameter: 2 bytes in FP16, 1 byte in 8-bit quantization, about 0.55 byte in 4-bit quantization including service data. KV cache: memory for the context; it grows linearly with context length and depends on the architecture. By our rule of thumb, for 8-billion-parameter models it is about 0.13 MB per token, and for 30–70 billion about 0.25–0.33 MB per token. Margin, by our rule: 10–20% for working buffers and the system.
Size guide in 4-bit quantization
8 billion parameters: about 5 GB of weights, runs on a laptop with 16 GB. 20–35 billion: 12–20 GB, one card with 24–32 GB or a Mac with 36–64 GB. 70 billion: about 40 GB, two 24 GB cards, one 96 GB card or a Mac with 64–128 GB. 235 billion: about 130 GB, a Mac Studio with 256–512 GB or several server cards. 671 billion: about 405 GB as measured for the 4-bit version, a Mac Studio with 512 GB, where such a model produces 6.21 tokens per second.
A real example
A cloud provider’s team fit two models on a 24 GB card with a context of 65,536 tokens: the text model’s weights took 13.0 GB, its KV cache 0.9 GB, the vision model’s weights 6.1 GB and its KV cache 2.4 GB, for a total of 22.4 GB out of 24. A margin of 1.6 GB was enough for single-user mode.
What this means when you buy
Memory matters more than speed: a model that does not fit does not work at all, while a slow one works. That is why people value unified memory of up to 512 GB in a Mac, and 24, 32, 80 and 96 GB in graphics cards. Several users at once multiply the KV cache: 10 parallel dialogs of 8,000 tokens on a 32-billion model need about 20 GB more.