MMOOZZGG.com

Glossary

What is a context window

In short

The context window is the amount of text, counted in tokens, that a language model sees at one time: instructions, documents, the question and its own answer.

Language models · context window

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine looking at a long painting through a small window cut in a sheet of paper. You see only the part inside the window, and you have to remember the rest or move the sheet. A neural network has such a window for text too. If a story is too long, the start drops out of view, and it forgets about it.

In plain words

A model does not remember a conversation by itself. Each time, everything is passed to it again: the instruction, the message history, attached files and the new question. The limit of this volume is called the window. It is measured in tokens, pieces of words, and the model’s answer also takes up space in it. When a conversation grows, old messages are cut, and the model stops taking them into account. For example, you discussed delivery terms at the start of a long chat, and an hour later it answers as if that conversation never happened.

For business

The window size decides how much material the model handles in one request: one contract or a whole archive of correspondence. A large window is handy, but every token you send is paid for, and a request of a hundred pages costs noticeably more than a short one. Research also shows that models find a needed fact in the middle of a long text worse than at the start or end. So for work with a knowledge base, teams usually do not load everything at once but find a few relevant passages through RAG. Before rollout, check on your own documents whether the model loses details on full loading and what such a request costs. These calculations are part of a pilot.

For a specialist

The context window is the maximum number of tokens a model takes into account at once, including input tokens and those already generated. In the transformer of Vaswani and co-authors (2017), the complexity of self-attention per layer is O(n²·d), that is, it grows quadratically with the length n, and during generation the KV cache (key-value cache) of each token takes up video memory. The window is extended by interpolating positional encodings (RoPE scaling) and by sliding window attention and sparse attention, and the memory for the KV cache in serving is saved by paged management with PagedAttention (Kwon and co-authors, 2023). The stated length is not the useful length: in “Lost in the Middle” (Liu and co-authors, 2023), accuracy dropped noticeably when the needed fact was in the middle of the input, even for long-context models. Useful length is tested with needle-in-a-haystack tests and with multi-step reasoning tasks over a long document.

Example

A manager loaded three years of correspondence with a customer into a chat and asked about a discount from the first email. The correspondence did not fit, the start was cut off, and the model answered that it knew nothing about a discount.

Related terms

Questions and answers

What happens if the text does not fit in the context window? The service either returns an error or cuts part of the text, usually the start of the conversation. The model answers only from what is left and gives no warning about the loss.
Does a larger context window mean higher quality? Not always. A long input costs more and runs slower, and the model finds details in its middle worse.

It is often more accurate to give it a few relevant passages than the whole archive.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us