Apple Silicon
Community benchmarks of llama.cpp on a 7–8 billion parameter model in 4-bit quantization, table updated on 25 August 2026: M4 Pro with 20 GPU cores gives 50.7 tokens per second, M4 Max with 40 cores 83.1, M3 Ultra with 80 cores 92.1, and M5 Max with 40 cores 119.9. M5 Max processes a long prompt at 3,220 tokens per second against 886 on M4 Max: the new generation reads context 3.6 times faster. A 671-billion-parameter model on a Mac Studio M3 Ultra with 512 GB gives 6.21 tokens per second and takes almost 15 minutes to process an 8,000-token prompt.
Graphics cards
Benchmarks from May 2024 on an 8-billion model in 4-bit quantization: RTX 4090 gives 127.7 tokens per second, A100 138.3, H100 144.5 and M2 Ultra 76.3. A 70-billion model in 4 bits: two RTX 4090 give 19.1, A100 22.1, H100 25.0 and M2 Ultra 12.1. A single user gets only a little more from a data center card than from a gaming card; the strength of data center cards is memory size and handling many parallel requests.
How much you need
A chat with a person: by our rule of practice, 15–20 tokens per second reads like live typing, and above 50 the difference is no longer visible. An agent that calls tools: prompt processing time matters more than generation, because the context with documents is read again at every step; here M5 Max and data center cards win. Batch processing of documents at night: count total output, not delay. The personal agent of a bank’s IT executive runs on a Mac at about 70 tokens per second.
How to test before you buy
Rent the card you need by the hour (as of 30 September 2026, from about $0.50 an hour for an RTX 4090 and from $3 for an H100) and run your own 50 requests, or borrow a Mac with the right memory from someone you know. A test on your own model and your own context is more accurate than any table.