MMOOZZGG.com

Glossary

What is tokens per second?

In short

Tokens per second (TPS) is the speed at which a neural network writes its answer: how many pieces of text it produces in one second.

Infrastructure · tokens per second

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine you are writing a dictation. One student writes one word a second, another writes three. The second one finishes first, and the teacher sees the whole text sooner. A smart assistant also writes its answer piece by piece, one after another. If it writes fast, the answer appears almost at once. If it writes slowly, you have to wait, as if someone were typing with one finger.

In plain words

A neural network does not output the whole answer at once. It writes it token by token, where a token is a piece of a word. The speed of this writing is measured in such pieces per second. If it is low, the text crawls across the screen, and you wait minutes for a long report. The speed depends on the model size, the graphics card and how many people use the system at the same time. There is also a second number: how long you wait for the first word of the answer. For example, the same model writes a letter at different speeds on a powerful server and on a laptop, even though the text is the same.

For business

This number decides how many employees or customers one graphics card can serve without a queue. Tell two speeds apart: how fast the answer comes for one person, and how many tokens the system produces in total for everyone. As the number of parallel requests grows, total throughput rises until the card is fully loaded, while the speed for each user falls. So spec-sheet figures for a single user say little about how a team of one hundred people will work. For chat, time to first token matters more; for batch processing of documents, total throughput does. The cost of an answer is also linked to speed: the more tokens a server produces per hour, the cheaper each one is. Measure speed on a test setup with your own requests and your own number of users.

For a specialist

The tokens per second (TPS) metric describes generation throughput and comes in two kinds. By the definitions in the NVIDIA NIM documentation on LLM benchmarking, system TPS is the total number of output tokens of all parallel requests divided by the time from the first request to the last response. TPS per user is the response length divided by the end-to-end latency of the request. Related metrics are time to first token (TTFT), inter-token latency (ITL, also called TPOT) and requests per second. As concurrency grows, system TPS rises until the GPU is saturated and may then fall, while TPS per user drops. Single-stream decoding is usually limited by memory bandwidth, since the weights are read again for every token. That is why quantization and speculative decoding raise speed, while the prefill of a long prompt increases TTFT.

Example

A company chooses a server for a legal team’s assistant. In a test with one user, the model writes fast, but with twenty parallel requests, each answer slows down several times. The team adds a second graphics card to keep the wait acceptable.

Related terms

Questions and answers

How many tokens per second do you need for comfortable work? It depends on the task. For chat, the answer should start quickly and run no slower than a person reads.

For batch processing of documents, the total throughput of the system matters. Figures for specific hardware and models are collected on the page about local model speed.

Why does speed drop when there are many users? The graphics card shares compute and memory among all requests. The total flow of tokens grows up to the hardware limit, but each user gets a smaller share, so their answer comes more slowly.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us