Glossary
What is LLM latency
In short
LLM latency is the delay between a request to a model and its answer: how long you wait for the first word and how long the whole answer takes.
Language models · latency
MMOOZZGG.com editorsChecked: 2 min read
For a child
You ask your mom what’s for dinner, and she doesn’t answer right away: first she thinks, then she starts to speak. The time from your question to her first word is the delay. If you wait too long, it feels like nobody heard you. Adults work hard so that computer helpers answer fast, without such pauses.
In plain words
When you write a question to a chatbot, some time passes before the first word appears, and then more time before the whole answer is typed. These pauses are called latency. Latency has several parts: the road of the request to the server, waiting in a queue, the model reading your text, and printing the answer piece by piece. A long question takes longer to read, and a long answer takes longer to print. In a text chat a pause of a couple of seconds is fine, but in a phone talk with a voice robot even a second of silence sounds awkward.
For business
For customer scenarios speed matters as much as accuracy. A voice agent with a long pause annoys the caller, and a website chat with long waiting loses the visitor. The delay depends on the model size, the length of the prompt and the answer, the server load and the distance to the data center: a foreign cloud adds travel time. A smaller model for simple steps helps, as do short instructions, streaming the answer as it is generated, and caching the repeated start of the prompt. Measure the tail of the distribution, not the average: one slow answer in twenty is more visible than all the fast ones. Put speed requirements into the specification of a voice agent and test them under load.
For a specialist
LLM service latency is split into metrics. Time to first token (TTFT), as NVIDIA defines it, includes queue time, prompt prefill and network delay. Inter-token latency (ITL), also called time per output token (TPOT), reflects only the decoding phase. End-to-end request latency equals TTFT plus generation time. Prefill is bound by compute and grows with prompt length; decoding is bound by memory bandwidth, because each step reads the model weights and the KV cache. Latency competes with throughput: a large batch raises tokens per second per server but makes each user’s answer longer. It is reduced with quantization, speculative decoding, grouped-query attention, prefix caching, and by placing prefill and decoding on separate nodes. Reports give the p50, p95 and p99 percentiles, not the average.
Do not confuse. Do not confuse it with latent space, the internal representations of a model: that is a different concept.
Example
A clinic’s voice agent answered with a noticeable pause, and patients asked again. After a move to a smaller model and streaming speech output, the first word came sooner and the talk became more natural.
Related terms
Questions and answers
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map