Glossary
What is AI inference?
In short
AI inference is the work of an already trained model when it receives a request and computes an answer, unlike training, where the model was only being tuned.
Language models · inference
MMOOZZGG.com editorsChecked: 1 min read
For a child
First you spend a long time learning the times tables, and then on a test you just use what you know: you see a problem and write the answer at once. Learning is over, and using it has begun. A neural network also has two times: when it learns and when it helps people. The second time, when it answers questions, is called by this clever word.
In plain words
A model is trained once, and it takes a long time. Everything that happens after that, when you ask a question and get an answer, is inference. The computer takes your text, runs it through billions of stored numbers and gives the answer piece by piece. Each request needs computing on a graphics card, so every answer has a price and a speed. For example, when a store’s chatbot answers a customer at two in the morning, the computing runs on the seller’s server or in the provider’s cloud.
For business
Inference is a permanent AI cost, unlike one-time training. In the cloud you pay for it by a per-token rate; on your own server you buy or rent graphics cards and pay for electricity and maintenance. The price depends on the size of the model, the length of requests and answers, the number of users at the same time and the speed you need. With a large and steady flow of requests, your own server can cost less than the cloud; with a small and uneven flow, paying for tokens is cheaper. Your own hardware also keeps data inside the company. The AI solution map compares these options in money for your volume of requests before you buy hardware.
For a specialist
Inference is a forward pass of a trained model without updating its weights. For an autoregressive LLM it has two phases: prefill, where the whole prompt is processed in parallel and the limit is compute, and decode, where tokens are generated one by one and memory bandwidth becomes the bottleneck. Kwon et al. (2023) proposed PagedAttention and the vLLM system: paged management of the KV cache with almost no memory waste raised throughput 2–4 times at the same latency compared with FasterTransformer and Orca. Leviathan, Kalman and Matias (2022) described speculative decoding: a small model proposes several tokens and a large model checks them in parallel, which made T5-XXL 2–3 times faster without changing the output. Other methods: continuous batching, weight quantization, tensor parallelism. Metrics: time to first token (TTFT), tokens per second, cost per million tokens.
Example
An online store launched an AI consultant on its own server. In the daytime it gets hundreds of requests an hour, and an engineer set up the system so that one graphics card serves several dialogs at once and an answer starts in under a second.
Related terms
Questions and answers
How much does inference cost?
Sources
Not sure whether your company needs this?
AI solutions map