MMOOZZGG.com

Glossary

What is AI inference?

In short

AI inference is the work of an already trained model when it receives a request and computes an answer, unlike training, where the model was only being tuned.

Language models · inference

MMOOZZGG.com editorsChecked: 1 min read

For a child

First you spend a long time learning the times tables, and then on a test you just use what you know: you see a problem and write the answer at once. Learning is over, and using it has begun. A neural network also has two times: when it learns and when it helps people. The second time, when it answers questions, is called by this clever word.

In plain words

A model is trained once, and it takes a long time. Everything that happens after that, when you ask a question and get an answer, is inference. The computer takes your text, runs it through billions of stored numbers and gives the answer piece by piece. Each request needs computing on a graphics card, so every answer has a price and a speed. For example, when a store’s chatbot answers a customer at two in the morning, the computing runs on the seller’s server or in the provider’s cloud.

For business

Inference is a permanent AI cost, unlike one-time training. In the cloud you pay for it by a per-token rate; on your own server you buy or rent graphics cards and pay for electricity and maintenance. The price depends on the size of the model, the length of requests and answers, the number of users at the same time and the speed you need. With a large and steady flow of requests, your own server can cost less than the cloud; with a small and uneven flow, paying for tokens is cheaper. Your own hardware also keeps data inside the company. The AI solution map compares these options in money for your volume of requests before you buy hardware.

For a specialist

Inference is a forward pass of a trained model without updating its weights. For an autoregressive LLM it has two phases: prefill, where the whole prompt is processed in parallel and the limit is compute, and decode, where tokens are generated one by one and memory bandwidth becomes the bottleneck. Kwon et al. (2023) proposed PagedAttention and the vLLM system: paged management of the KV cache with almost no memory waste raised throughput 2–4 times at the same latency compared with FasterTransformer and Orca. Leviathan, Kalman and Matias (2022) described speculative decoding: a small model proposes several tokens and a large model checks them in parallel, which made T5-XXL 2–3 times faster without changing the output. Other methods: continuous batching, weight quantization, tensor parallelism. Metrics: time to first token (TTFT), tokens per second, cost per million tokens.

Example

An online store launched an AI consultant on its own server. In the daytime it gets hundreds of requests an hour, and an engineer set up the system so that one graphics card serves several dialogs at once and an answer starts in under a second.

Related terms

Questions and answers

How is inference different from training? In training, the model changes its weights on examples; this takes long and needs a lot of computing. In inference, the weights do not change and the model only computes an answer to a request, but this repeats for every request and over time becomes the main cost.
How much does inference cost? In the cloud, the price is per million input and output tokens, and it depends on the model. On your own server, the cost is the graphics cards, electricity and maintenance divided by the volume of requests.

Compare the options on your real flow of requests.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us