Glossary
What is tokens per second?
In short
Tokens per second (TPS) is the speed at which a neural network writes its answer: how many pieces of text it produces in one second.
Infrastructure · tokens per second
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine you are writing a dictation. One student writes one word a second, another writes three. The second one finishes first, and the teacher sees the whole text sooner. A smart assistant also writes its answer piece by piece, one after another. If it writes fast, the answer appears almost at once. If it writes slowly, you have to wait, as if someone were typing with one finger.
In plain words
A neural network does not output the whole answer at once. It writes it token by token, where a token is a piece of a word. The speed of this writing is measured in such pieces per second. If it is low, the text crawls across the screen, and you wait minutes for a long report. The speed depends on the model size, the graphics card and how many people use the system at the same time. There is also a second number: how long you wait for the first word of the answer. For example, the same model writes a letter at different speeds on a powerful server and on a laptop, even though the text is the same.
For business
This number decides how many employees or customers one graphics card can serve without a queue. Tell two speeds apart: how fast the answer comes for one person, and how many tokens the system produces in total for everyone. As the number of parallel requests grows, total throughput rises until the card is fully loaded, while the speed for each user falls. So spec-sheet figures for a single user say little about how a team of one hundred people will work. For chat, time to first token matters more; for batch processing of documents, total throughput does. The cost of an answer is also linked to speed: the more tokens a server produces per hour, the cheaper each one is. Measure speed on a test setup with your own requests and your own number of users.
For a specialist
The tokens per second (TPS) metric describes generation throughput and comes in two kinds. By the definitions in the NVIDIA NIM documentation on LLM benchmarking, system TPS is the total number of output tokens of all parallel requests divided by the time from the first request to the last response. TPS per user is the response length divided by the end-to-end latency of the request. Related metrics are time to first token (TTFT), inter-token latency (ITL, also called TPOT) and requests per second. As concurrency grows, system TPS rises until the GPU is saturated and may then fall, while TPS per user drops. Single-stream decoding is usually limited by memory bandwidth, since the weights are read again for every token. That is why quantization and speculative decoding raise speed, while the prefill of a long prompt increases TTFT.
Example
A company chooses a server for a legal team’s assistant. In a test with one user, the model writes fast, but with twenty parallel requests, each answer slows down several times. The team adds a second graphics card to keep the wait acceptable.
Related terms
Questions and answers
How many tokens per second do you need for comfortable work?
Sources
Not sure whether your company needs this?
AI solutions map