Glossary
What is a KV cache
In short
The KV cache is the memory in which a language model keeps intermediate results (attention keys and values) for tokens it has already processed, so it does not recompute them for every new token.
Language models · KV cache
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine you solve a long problem step by step and write each answer along the way on a sheet of paper. At the next step you don’t count everything again from the start, you look at your notes. The sheets keep piling up, and soon your desk is full. A neural network also keeps such notes when it writes an answer, and they need space too.
In plain words
A model writes an answer one piece at a time, and each time it looks back at the whole earlier text. Without a trick, it would have to digest your question and everything already written again at each step. So the results of processing the old pieces go into a separate memory, and the model takes them from there ready-made. The answer gets much faster, but the memory grows with every word. A long document and many people chatting at once fill it fastest, and the graphics card runs out of space before it runs out of computing power.
For business
The cache decides how many users and how long documents your server can handle. Model weights take video memory once, while the cache takes it for every active dialog, in proportion to the context length. So a server where the model works fine for one employee starts to refuse or queue requests at twenty parallel requests with long contracts. When you size hardware, count the parallel users, the typical length of a request and an answer, and whether the inference program supports cache compression. The same idea is behind prompt caching at cloud providers. Calculate the memory for the real load before you buy a server.
For a specialist
In autoregressive decoding, each new token computes attention to all the earlier ones. The KV cache stores the key and value tensors of all layers for positions already processed, so at each step only the projections of the new token are computed. Pope and colleagues (2022) use the name for the keys and values of each layer that must stay in memory for the whole decoding period, and stress that, unlike the weights, it is unique to each sequence in a batch. Its size per sequence is 2 × number of layers × number of KV heads × head dimension × length × bytes per element. Architecture can shrink it: multi-query attention (Shazeer, 2019) shares one pair of keys and values among all heads, and grouped-query attention (Ainslie and colleagues, 2023) uses an in-between number of groups. At the system level, teams use paged storage (PagedAttention), cache quantization, eviction of old tokens and reuse of shared prefixes between requests.
Example
A company runs a model on its own server for its lawyers. While the documents are short, everything works. When five lawyers upload hundred-page contracts at the same time, the KV cache fills the video memory, and the inference program starts to queue requests.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map