Glossary
What is a small language model?
In short
A small language model (SLM) is a compact language model with a few billion parameters or fewer that runs on an ordinary server, a laptop or a phone.
Language models · small language model
MMOOZZGG.com editorsChecked: 1 min read
For a child
A big encyclopedia knows almost everything, but it is heavy and stays in the library. A pocket dictionary knows less, but it is always in your school bag and opens in a second. If you need to check a word quickly, the dictionary is enough. Small neural networks are like that dictionary: they fit even in a phone.
In plain words
Large neural networks run in powerful data centers, and every question flies to them over the internet. A small model is many times more compact, so you can run it right on a computer or a phone. It knows fewer facts and handles tangled tasks worse, but it answers fast, without the internet, and does not send your texts outside. It is good at narrow, repeated jobs: sorting an email, filling in a form, summing up a document. It is like a calculator in your pocket: not a supercomputer, but enough to add up your shopping.
For business
Compact models pay off where there are many requests and the task is narrow: sorting requests, pulling payment details from invoices, steps of an AI agent such as picking the right tool. They run on one inexpensive graphics card or right at the workstation, and personal data does not leave the company’s closed environment. Costs are the server, tuning for your task and support. Risks: on open questions such a model makes mistakes and invents things more often, so it is not used where broad knowledge is needed. The usual setup is mixed: a small model takes the flow of routine, and complex requests go to a large one. The choice between a local and a cloud option depends on the volume of requests and the rules for data.
For a specialist
A small language model (SLM) has no strict size limit: the term usually means models with a few billion parameters that are fit for inference on one GPU, on a CPU or on-device. Quality at a small size comes from selecting and synthesizing training data, distillation from larger models and long training on a large number of tokens. An example from the Phi-3 technical report (2024): phi-3-mini has 3.8 billion parameters, was trained on 3.3 trillion tokens, scores 69% on MMLU and fits in a phone. Belcak et al. (2025) argue that for agentic systems, where a model performs a few narrow, repeated operations, SLMs are powerful enough and cheaper, and general LLMs should be called only where needed in heterogeneous systems. For deployment, such models are usually also quantized.
Example
An accounting department gets hundreds of invoices a day. A compact model on the company’s server takes the payment details, the amount and the date from each one and enters them into the accounting system, and it passes disputed invoices to a person.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map