MMOOZZGG.com

Basics

Tokens per second for an LLM: what speed you need

3 min read10 sources

Tokens per second for an LLM decides whether a person waits for the answer or not. A token is a piece of a word: for one major Russian cloud model the average is 3–4 characters with spaces, so a page of 1,800 characters is about 450–600 tokens; other models count differently. Below are real, dated benchmarks on Mac and graphics cards, and a rule for how much speed each task needs.

On this page · 4 sections

Apple Silicon

Community benchmarks of llama.cpp on a 7–8 billion parameter model in 4-bit quantization, table updated on 25 August 2026: M4 Pro with 20 GPU cores gives 50.7 tokens per second, M4 Max with 40 cores 83.1, M3 Ultra with 80 cores 92.1, and M5 Max with 40 cores 119.9. M5 Max processes a long prompt at 3,220 tokens per second against 886 on M4 Max: the new generation reads context 3.6 times faster. A 671-billion-parameter model on a Mac Studio M3 Ultra with 512 GB gives 6.21 tokens per second and takes almost 15 minutes to process an 8,000-token prompt.

Graphics cards

Benchmarks from May 2024 on an 8-billion model in 4-bit quantization: RTX 4090 gives 127.7 tokens per second, A100 138.3, H100 144.5 and M2 Ultra 76.3. A 70-billion model in 4 bits: two RTX 4090 give 19.1, A100 22.1, H100 25.0 and M2 Ultra 12.1. A single user gets only a little more from a data center card than from a gaming card; the strength of data center cards is memory size and handling many parallel requests.

How much you need

A chat with a person: by our rule of practice, 15–20 tokens per second reads like live typing, and above 50 the difference is no longer visible. An agent that calls tools: prompt processing time matters more than generation, because the context with documents is read again at every step; here M5 Max and data center cards win. Batch processing of documents at night: count total output, not delay. The personal agent of a bank’s IT executive runs on a Mac at about 70 tokens per second.

How to test before you buy

Rent the card you need by the hour (as of 30 September 2026, from about $0.50 an hour for an RTX 4090 and from $3 for an H100) and run your own 50 requests, or borrow a Mac with the right memory from someone you know. A test on your own model and your own context is more accurate than any table.

Questions and answers

Why does the model think for a long time before answering on a long document?

This is prompt processing: the model reads the whole context before the first token. On a Mac Studio M3 Ultra, an 8,000-token prompt to a 671-billion model took 888 seconds.

Sources and fact-checking

Checked by the editors on 29 September 2026.

  1. llama.cpp, performance on Apple Silicon, table updated 25 August 2026 (github.com)
  2. GPU-Benchmarks-on-LLM-Inference, Llama 3 8B and 70B, May 2024 (github.com)
  3. Hardware Corner, DeepSeek 671B on Mac Studio M3 Ultra 512 GB, 27 March 2025 (hardware-corner.net)
  4. Tproger, cloud GPUs for ML in 2026, 21 May 2026 (tproger.ru)
  5. Developer documentation of a major Russian AI model, token count: 3–4 characters per token on average, 17 July 2026 (developers.sber.ru)
  6. GPU Index, RTX 4090 rental in Russia: 173 offers, from about $0.40 per hour (checked 30 September 2026) (gpuindex.ru)
  7. GPU Index, H100 rental in Russia: 41 offers, from $3 per hour (checked 30 September 2026) (gpuindex.ru)
  8. Digital Platforms, case no. 2447, October 2025: a large Russian bank (platforms.su)
  9. Digital Platforms, case no. 3855, 2 September 2026: the technology group of a large Russian bank (platforms.su)
  10. Digital Platforms, case no. 4067, 2 September 2026: Alfa-Bank, IT executive (platforms.su)

Found a mistake? Tell us

Publications

Mike Mozg

Agentic AI use cases at AT&T: nearly 500 processes now run on agents

AT&T no longer uses AI only to help staff work faster: it is rebuilding whole business processes around agents. SiliconANGLE reported on 14 November 2026 that nearly 500 agentic AI use cases at AT&T are already in production. Here is how the company rebuilds them, where a human still checks the agent, and what other large companies can take from it.

News4 min

Cases on this topic: 3

  • Market case · Russia

    A large Russian bank in-house LLMs solve 45% of requests; support costs estimated down at least 25% a year

    The bank put its own family of language models into its chatbot for LLM customer support: about 45% of customer requests are solved without an operator. In October 2025 it launched AI copilots for first-line operators, and more than 90% of support staff use them every day. The models run inside the bank’s network perimeter, so data does not leave the bank.

    Banks and finance

    source: platforms.su

    Read the case
  • Market case · Russia

    A large Russian financial group AI financial assistant on in-house language models, beta since 20 July 2026

    The group built an AI financial assistant for tasks linked to money and shopping, on its own family of large language models. To avoid made-up answers, the system relies on verified data sources, not only on what the model has learned. The beta was presented on 20 July 2026, and the full launch is planned for the autumn.

    Banks and finance

    2 September 2026 · source: platforms.su

    Read the case
  • Market case · Russia

    Alfa-Bank, IT head local agent on the Qwen 3.6–35B-A3B model: about 70 tokens per second on a Mac

    A local AI agent on Mac does autonomous work on a model with 35 billion parameters, about 3 billion of them active per token. It runs through LM Studio at about 70 tokens per second. The agent first ran in Docker on the Mac, then moved to a virtual machine, and the author calls that move a turning point for ease of use.

    Banks and finance

    2 September 2026 · source: platforms.su

    Read the case
Contact us