MMOOZZGG.com

Glossary

What is VRAM?

In short

VRAM (video memory) is the memory of a graphics card that holds a neural network’s weights and intermediate data while it runs; its size decides which model fits on the card.

Infrastructure · VRAM

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine the desk where you do your homework. If the desk is big, your textbook, notebook, and pencils all fit on it and are close at hand. If it is small, you keep running to the shelf for the right book, and the work goes slowly. A graphics card has a desk like that too. The bigger it is, the bigger the helper you can lay out on it whole.

In plain words

A graphics card has its own memory, separate from the ordinary RAM. Before a neural network starts work, it is loaded into this memory whole: all the numbers that make up the model. The model also needs space for the text of the conversation it keeps in view while it answers. If there is not enough memory, the model either will not start, or part of it has to stay in the computer’s ordinary memory, and answers become much slower. So when you choose a graphics card for a neural network, look first at the memory size in gigabytes, not at the speed.

For business

VRAM size shapes the budget for local AI more than any other parameter. It sets which model you can run, how long a document it can read at once, and how many employees can use it at the same time. Cards with more memory cost disproportionately more, so models are often compressed to fit on one card instead of two. A typical mistake is to count only the model size and forget long requests and parallel users. The server is bought, and under real load the memory runs out. Calculate the size for your own scenarios with a reserve for growth, and test it on a trial setup before you buy.

For a specialist

Video memory (video RAM, VRAM) is the memory on the accelerator board: GDDR in consumer cards and some server cards, HBM in high-end server accelerators, with bandwidth far above system DRAM. During LLM inference, it holds the weights, the attention key-value cache (KV cache) for each request, and the activations. Kwon et al. (2023) give an example: a 13-billion-parameter model on an A100 with 40 GB, where about 65% of the memory goes to the weights (26 GB) and almost 30% to the KV cache, which grows with the context length and the number of parallel requests. Wasteful cache allocation leads to fragmentation, so the authors proposed PagedAttention and the vLLM engine. The size of the weights is estimated as the number of parameters times the bytes per parameter; quantization to 4 or 8 bits cuts it several times. If the model does not fit, llama.cpp moves only some layers to the GPU (offloading), and the CPU computes the rest, at the cost of speed.

Example

A company chooses a graphics card for a model that answers employees from a knowledge base. The compressed model takes about half of the card’s memory, and the rest goes to long internal rules in requests and to about ten users at the same time.

Related terms

Questions and answers

How much VRAM does a neural network need? It depends on the model size, the number format, and the length of requests. A rough estimate: the number of parameters times the bytes per parameter, plus a reserve for the context cache and for parallel users.
Can I run a model if there is not enough VRAM? You can compress the model with quantization or move some layers to ordinary memory. The model will run, but answers will be slower, and the compressed version may lose some quality.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us