Glossary
What is prompt caching
In short
Prompt caching is a technique in which the model provider saves the already processed start of a request and takes it from memory when it repeats: the answer arrives faster and costs less.
Language models · prompt caching
MMOOZZGG.com editorsChecked: 1 min read
For a child
Every morning you pack your school bag with the same textbooks. You can make the pile in advance and just take it, adding only today’s notebook. A neural network can also remember the part of the task that does not change and not read it again every time. So it answers faster.
In plain words
Many requests to a model start the same way: a long instruction, a company handbook, a description of the tools, and only the question at the end changes. Each time the model has to reread that whole start. A cache solves this: the processed start is saved for a short time, and the next request with the same start uses the ready result. The model spends effort only on the new part. There is one condition: the start must match exactly, down to the character.
For business
For a company with an assistant or an agent, this is a direct saving. According to Anthropic’s documentation, reading from the cache costs 2.5 to 10% of the base input token price, depending on the model, and writing to a five-minute cache costs 1.25 times the normal input. OpenAI turns caching on by default and gives a discount of up to 95% on reused input. The gain depends on how requests are built: the fixed part goes first and the changing part last, otherwise the cache does not work. The risk is small but real: the cache lives for minutes and can clear before a rare request comes. Before you set a budget, compare the prices of different models with the cache taken into account.
For a specialist
Prompt caching reuses the computed attention states, the KV cache, for a shared prefix of requests. In Prompt Cache, Gim and coauthors (2023) proposed describing repeated fragments (system messages, templates, documents) as modules and storing their states. Time to first token fell 8 times on a GPU and 60 times on a CPU with no loss of accuracy. In an API, the cache works on an exact match of the prefix. At Anthropic, the tools, the system prompt and the messages up to the block marked cache_control are cached in that order. The default lifetime is 5 minutes and is extended for free on every hit, and one hour is available for an extra fee. The minimum length of a cached prefix depends on the model, from 512 to 4096 tokens. At OpenAI the cache is on by default, the minimum for GPT-5.6 and newer is 1024 tokens, and the prefix is kept for 30 minutes after its last use. The stable part goes first, and timestamps and user IDs go to the end.
Example
A support team sends the model the same instruction and a large handbook with every question. After the cache is turned on, this part is billed at a lower price, and the answers come noticeably faster.
Related terms
Questions and answers
How much can you save with prompt caching?
Does the cache affect the model’s answers?
Sources
Not sure whether your company needs this?
AI solutions map