MMOOZZGG.com

Glossary

What is an AI API rate limit

In short

An AI API rate limit is a cap on how many requests and tokens a program can send to a cloud neural network per minute or another period.

Infrastructure · rate limit

MMOOZZGG.com editorsChecked: 2 min read

For a child

In the school cafeteria, one cook serves the food. If the whole class runs up at once, the cook can’t keep up, so a helper lets kids through in turns: as many as the cook can feed. The rest wait a minute and come back. Smart helpers on the internet have a helper like that too. It makes sure nobody takes the whole cook and everyone gets a plate.

In plain words

A neural network service serves thousands of clients at once, and its capacity is not endless. So every account has a ceiling: how many requests per minute it can send and how many tokens it can read and write. While you stay under the ceiling, everything works as usual. If a program goes over it, the service refuses with an error and often says after how many seconds you can retry. For example, at peak hour a support bot got twice as many questions as usual, and some answers came a few seconds late until the queue cleared.

For business

These ceilings decide how many customers your AI agent can serve in the peak hour, so count them before launch, not after the first complaint. Providers raise the ceiling as your usage history grows: for example, at Anthropic the limits depend on the account’s usage tier, and each tier also has a monthly spending cap. A typical failure: a sale or a mailing brings a spike, the agent hits the limit, and customers wait. What to check: the values for the model you need at your tier, how the system behaves on a refusal (a queue, a retry, a backup model), caching of repeated instructions, and separate quotas for different departments. Your own model on a server removes outside limits, but then your hardware sets the speed. Dependence on a provider’s rules is covered in AI risk assessment.

For a specialist

Rate limiting protects a service from overload and abuse and shares capacity between clients. In HTTP, going over the limit is marked by code 429 Too Many Requests, introduced by Nottingham and Fielding in RFC 6585 (April 2012); the response may carry a Retry-After header with the pause before a retry. Language model APIs keep several counters: requests per minute (RPM), and input and output tokens per minute (ITPM, OTPM). Anthropic uses the token bucket algorithm: capacity refills continuously up to a maximum instead of resetting once a minute, so over short intervals 60 RPM can act as one request per second. For most Claude models, tokens read from the prompt cache do not count toward ITPM, so caching raises the effective throughput. A client should retry with exponential backoff and read the headers that show the remaining quota.

Example

An online store sends a mailing to its whole customer base, and the AI agent must answer everyone who writes in the first hour. The team counts the peak requests per minute in advance, puts incoming messages in a queue and turns on caching of the agent’s shared instruction, so it does not hit the token limit.

Related terms

Questions and answers

What should you do if a neural network answers with error 429? The error means you went over the cap on requests or tokens. Wait as many seconds as the Retry-After header says and repeat the request; in code, retries use a growing pause.

If the error keeps coming back, ask the provider to raise your quota or spread the load.

How is a rate limit different from a spending limit? The first limits speed: how many requests and tokens pass per minute. The second limits money per month, and once it is used up, requests fail until the next period starts or the cap is raised.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us