MMOOZZGG.com

Glossary

Glossary: local AI terms in plain words

4 min read9 sources

Local AI terms that appear on the pages of this section, each with a short explanation and a reference number where one exists. The order goes from the model to the hardware and the law.

On this page · 4 sections

Model

Token: a unit of text for the model, a piece of a word; according to the developer docs of a large Russian bank, its model averages 3–4 characters per token, so a 10-page document of 1,800 characters a page is about 4,500–6,000 tokens. Parameters: the numbers inside a model, written as 8B or 70B, billions; the more there are, the smarter and heavier the model. Active parameters: in models with a sparse architecture only part of the parameters work on each token, for example 3 billion of 35, hence a speed of 70 tokens per second on a Mac. Context: how many tokens the model sees at once; the longer it is, the more memory the cache needs, for example an agent of a large Russian tech company worked with a context of 65,536 tokens. Open weights: the model files are published, so you can download the model and run it yourself; the Apache 2.0 license allows commercial use. Fine-tuning: training an open model on your own data, which needs GPUs and time; an IT holding spent more than a year training a model on contracts. RAG, search in documents: the model answers from the fragments of your documents it found, without fine-tuning. Reasoning mode: the model writes intermediate steps before the answer, which is more accurate and costs more tokens.

Memory and speed

Quantization: compressing weights from 16 bits to 8 or 4; memory drops 2–3.6 times by the factors of our calculator, and you should check quality on your own requests; 70 billion parameters in 4 bits take about 40 GB. FP16: a 16-bit weight format, 2 bytes per parameter. KV cache: memory for the context, 0.13–0.33 MB per token by our calculator’s rule depending on model size, multiplied by the number of users. Unified memory: in Apple Silicon the processor and graphics share one memory of up to 512 GB, so a Mac holds models that do not fit into a graphics card. Video memory, VRAM: the memory on a graphics card, 24–96 GB; the model must fit in it completely. Tokens per second: the generation speed; one stream on an 8B model in 4 bits gives from 50.7 on an M4 Pro to 144.5 on an H100 by published measurements. Prompt processing: reading the context before the first token, 3,220 tokens per second on an M5 Max with a 7B model in 4 bits by the llama.cpp table. Layer offloading: part of the model sits in system memory when video memory is short; speed drops noticeably.

Hardware and running the model

Inference engine: a program that runs the model: llama.cpp, LM Studio, Ollama, vLLM. Batched inference: handling dozens of requests at once on a server card, with a higher total speed than one stream. Server card: A100, H100, RTX PRO 6000 with 40–96 GB and memory error correction, for a data center. Gaming card: RTX 4090, RTX 5090 with 24–32 GB, for a workstation. Rack unit, colocation: a place in a data center rack, from $90 a month with 300 W. Unified model interface: a way to connect agents so that the model can be changed in the settings. MCP: the protocol through which an agent connects tools: databases, APIs, documents; a telecom operator requires it in its tender. Router: a part of a hybrid setup that decides where a request goes: to the local model or to the cloud.

Law

Sovereign model: under 243-FZ, from 1 March 2027, a model that a Russian legal entity develops and can fully reproduce, including training, with answers and data storage in Russian data centers. National model: allows open foreign components if the servers and data are in Russia. Closed perimeter: a network with no internet access where the model and the data live. Local deployment, on-premises: the model on your own servers, and the network may be open. Processing agreement: a document under 152-FZ, Russia’s personal data law, under which a provider processes personal data on your behalf; most foreign providers do not have one. Anonymization: replacing names and numbers with labels before sending data to the cloud; under 152-FZ, after it you cannot tell whose data it is without additional information; if the link between a label and a person is easy to restore, the data may remain personal.

Sources and fact-checking

Checked by the editors on 29 September 2026.

  1. llama.cpp, performance on Apple Silicon, table updated 25 August 2026 (github.com)
  2. GPU-Benchmarks-on-LLM-Inference, Llama 3 8B and 70B, May 2024 (github.com)
  3. HOSTKEY, colocation in Moscow, 2026 (hostkey.ru)
  4. CNews, which AI models will get sovereign status, 28 September 2026, a piece marked “advertising” (corp.cnews.ru)
  5. Anti-Malware, 152-FZ and neural networks, 17 June 2026 (anti-malware.ru)
  6. iXBT, a large Russian bank released a 32B open model (Apache 2.0), 18 July 2025 (ixbt.com)
  7. Developer docs of a large Russian bank, token counting for its model: 3–4 characters per token on average, 17 July 2026 (developers.sber.ru)
  8. Federal Law No. 243-FZ of 26 July 2026, text on ConsultantPlus (consultant.ru)
  9. Federal Law No. 152-FZ “On Personal Data”, text on ConsultantPlus (consultant.ru)

Found a mistake? Tell us

Publications

Contact us