MMOOZZGG.com

Basics

How much memory does an LLM need: simple math and real hardware

2 min read7 sources

How much memory does an LLM need, and will it fit? One formula decides: parameters times bytes per parameter, plus the context cache, plus a margin. Below are the formula, a size guide and proof from a real case where two models fit in 24 GB of video memory.

On this page · 4 sections

The formula

Weights: the number of parameters times the bytes per parameter: 2 bytes in FP16, 1 byte in 8-bit quantization, about 0.55 byte in 4-bit quantization including service data. KV cache: memory for the context; it grows linearly with context length and depends on the architecture. By our rule of thumb, for 8-billion-parameter models it is about 0.13 MB per token, and for 30–70 billion about 0.25–0.33 MB per token. Margin, by our rule: 10–20% for working buffers and the system.

Size guide in 4-bit quantization

8 billion parameters: about 5 GB of weights, runs on a laptop with 16 GB. 20–35 billion: 12–20 GB, one card with 24–32 GB or a Mac with 36–64 GB. 70 billion: about 40 GB, two 24 GB cards, one 96 GB card or a Mac with 64–128 GB. 235 billion: about 130 GB, a Mac Studio with 256–512 GB or several server cards. 671 billion: about 405 GB as measured for the 4-bit version, a Mac Studio with 512 GB, where such a model produces 6.21 tokens per second.

A real example

A cloud provider’s team fit two models on a 24 GB card with a context of 65,536 tokens: the text model’s weights took 13.0 GB, its KV cache 0.9 GB, the vision model’s weights 6.1 GB and its KV cache 2.4 GB, for a total of 22.4 GB out of 24. A margin of 1.6 GB was enough for single-user mode.

What this means when you buy

Memory matters more than speed: a model that does not fit does not work at all, while a slow one works. That is why people value unified memory of up to 512 GB in a Mac, and 24, 32, 80 and 96 GB in graphics cards. Several users at once multiply the KV cache: 10 parallel dialogs of 8,000 tokens on a 32-billion model need about 20 GB more.

Questions and answers

Can you run a model if memory is slightly short?

Some layers move to system memory or to disk, and speed drops several times over. It is better to take a lower quantization or a smaller model.

Sources and fact-checking

Checked by the editors on 29 September 2026.

  1. Hardware Corner, DeepSeek 671B on a Mac Studio M3 Ultra with 512 GB, 27 March 2025 (hardware-corner.net)
  2. llama.cpp, performance on Apple Silicon, table updated 25 August 2026 (github.com)
  3. Digital Platforms, case no. 2929, 2025: a large Russian IT holding (platforms.su)
  4. Digital Platforms, case no. 3475, 2 September 2026: NSIS, operator of an insurance information system (platforms.su)
  5. Digital Platforms, case no. 3661, 2 September 2026: Digital Sector and a client from IT infrastructure (platforms.su)
  6. Digital Platforms, case no. 4084, 2 September 2026: the cloud unit of a major Russian internet company (platforms.su)
  7. Digital Platforms, case no. 2874, 2025: LANIT (platforms.su)

Found a mistake? Tell us

Publications

Cases on this topic: 5

  • Market case · Russia

    A large Russian IT holding contract approval in 3 days instead of two weeks, document comparison costs 40% lower

    The legal department of the holding, more than 70 lawyers, trained its own language model on the company’s legal data for over a year. Contract approval fell from two weeks to three days. AI contract comparison of two versions of a document is five times faster than by hand.

    IT companies

    source: platforms.su

    Read the case
  • Market case · Russia

    NSIS, insurance information system operator requests handled 20 times faster on a local LLM, 1.5 times more requests with no new hires

    The operator of the national insurance information system deployed a local LLM, a large language model on its own servers, and built three uses on it. The first is automatic processing of requests with many fields. The second is a RAG service for technical support that answers from company documents. The third is AI agents in software development, including automatic code review.

    IT companies

    2 September 2026 · source: platforms.su

    Read the case
  • Market case · Russia

    Digital Sector and an IT infrastructure client AI contract analysis agent in the client’s closed environment; in the integrator’s own testing, agents freed 20% of team time

    The integrator built an AI agent for first-pass contract analysis inside the closed network of a client that designs and supplies IT infrastructure. The agent extracts and matches data, finds discrepancies and prepares the result for a specialist. The integrator also put agents into its own testing process, which freed 20% of the team’s working time per sprint.

    IT companies

    2 September 2026 · source: platforms.su

    Read the case
  • Market case · Russia

    VK Tech two models fit into 22.4 GB of 24 GB of video memory with a 65,536-token context

    The team built a local AI agent in a closed environment with Ollama and an agent layer, on a rented GPU server with one NVIDIA A30 card of 24 GB. The models were gpt-oss 20b (13 GB) for text and tools and qwen3-vl 8b (6.1 GB) for images; with the KV cache for a 65,536-token context, they used 22.4 GB.

    IT companies

    2 September 2026 · source: platforms.su

    Read the case
  • Market case · Russia

    LANIT requirements gathering 4 times faster, document search time 66% lower on a local LLM platform

    The group built a platform that runs large language models on the customer’s own infrastructure: a chat and complex pipelines for documents, audio and images. A subsidiary built its own model, trained further on Russian-language data. It pilots the model in CRM, SRM and HRM projects and in corporate chatbots.

    IT companies

    source: platforms.su

    Read the case
Contact us