MMOOZZGG.com

Glossary

What is a local LLM

In short

A local LLM is a neural network that runs on your own computer or server, not in a vendor’s cloud, so your data goes nowhere.

Infrastructure · local model

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you have your own encyclopedia at home. To find an answer, you don’t need to go to the library, wait in line and ask the librarian. The book is on your shelf, and nobody knows what you looked up in it. A local model is like a home encyclopedia: it lives on your computer and answers even without the internet.

In plain words

Usually people use a neural network through a website or an app. The question flies to another server, a model processes it there, and the answer comes back. With a local launch, everything happens on your hardware. You download a file with open weights, start it with a program such as Ollama or LM Studio, and chat with the model without the internet. Small models run on a laptop with a good graphics card or on a Mac; big ones need a server. In return you get privacy, but speed and quality are limited by your computer.

For business

Companies choose their own launch when data must not go outside: correspondence with customers, contracts, medical and financial data. The second reason is independence from a vendor who can switch off the service or raise prices. The cost structure differs from the cloud: instead of paying per token, you buy or rent a server and pay for electricity, maintenance and an engineer. With a small volume of requests the cloud is usually cheaper. With a constant high load, your own hardware can pay back. On some tasks open models are weaker than the leading cloud models, so first compare them on your real examples, and only then buy equipment.

For a specialist

A local model is an LLM or another neural network whose inference runs on the user’s device or server, with no call to an outside API. It rests on open-weights models and inference engines. The llama.cpp project aims to run LLMs with minimal setup on a wide range of hardware, including Apple silicon. It supports integer quantization from 1.5 to 8 bits and a hybrid CPU and GPU mode for models larger than video memory. For server load with parallel requests, teams use vLLM with PagedAttention. Post-training quantization makes a launch possible on affordable hardware: the GPTQ method (Frantar et al., 2022) compresses weights to 3–4 bits with little loss of accuracy, and for the first time fit a 175-billion-parameter model on one GPU. The trade-off is set by model size, weight precision and context length, and it is limited by memory size and memory bandwidth.

Example

A law firm checks contracts with a model that runs on a server in its own office. The contract texts do not leave the company network, and a lawyer gets a list of risky clauses in a few minutes.

Related terms

Questions and answers

What is a local neural network? It is the same thing: a neural network that you download and run on your own computer or server. It works without sending data to the cloud and often without the internet.
Can I run a neural network on an ordinary computer? Small models run on a laptop or a Mac with enough memory, especially in compressed form. Large models and many users at once need a server with a strong graphics card.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us