Glossary
What is cache augmented generation (CAG)?
In short
CAG (cache-augmented generation) is a method where all the needed documents are loaded into the model’s context in advance and its intermediate state is saved, so no search is needed at answer time.
Search and RAG · cache-augmented generation
MMOOZZGG.com editorsChecked: 1 min read
For a child
Before a test, you could run to the shelf for a textbook for every question. Or you could put three thin books on your desk in advance and keep them open. Then you find the answer at once, with no running. But if you have a whole pile of books, they will not fit on the desk. Cache augmented generation is about keeping short books right in front of you.
In plain words
A standard RAG system searches the database for suitable pieces for every question and only then answers. CAG is simpler. If there are only a few documents, they are loaded into the model in full, once, in advance. The model reads them and saves the result of this work in a special cache. When a question comes in, it is simply added to what the model has already read, and the model answers at once. No search means no error when the search brings back the wrong piece. For example, an assistant that knows the vacation policy and two internal orders can work like this all day.
For business
This approach pays off when the knowledge base is small and rarely changes: a set of rules, a price list, the rules of one product. You do not need to build a vector database, choose a search model, or tune the chunking, so the system is simpler and answers faster. The limit is strict: all documents must fit into the context window, and for much larger bases the method becomes impractical. The cache lives in GPU memory, and on your own servers that is a real cost item. Each time the documents change, the cache is computed again. Before you choose, count the size of the text in tokens and how often it is edited; GPUs for such a cache are sized separately.
For a specialist
Cache-augmented generation was proposed by Chan, Chen, Cheng, and Huang (2024) in a paper titled “Don’t Do RAG”. Instead of retrieval at query time, all relevant documents are fed in advance into a model with a long context, and its inference state is saved: the model computes a key-value cache (KV cache). At answer time only the question is added to the cache, and the documents are not processed again. The authors name the gains: no retrieval delay, no document selection errors, and a simpler architecture. The experiments used SQuAD 1.0 and HotPotQA with the Llama 3.1 8B model, which accepts up to 128K tokens. The paper itself states the limit: the method suits knowledge sources of manageable size and becomes impractical for much larger sets. A close technique in cloud APIs is called prompt caching: the provider stores the processed repeated prefix of a request. You can combine CAG and RAG: keep a stable core of knowledge in the cache and search for the rest.
Example
The HR department loads the internal work rules and the vacation policy into an assistant. The cache is computed in the morning, and all day employees get answers with no search in the database. After the policy is edited, the cache is computed again.
Related terms
Questions and answers
How is CAG different from RAG?
Sources
Not sure whether your company needs this?
AI solutions map