Glossary
What is a long context LLM?
In short
Long context is the ability of a language model to take in and use a very large amount of text in one request: thick documents, a long correspondence or a whole code repository.
Language models · long context
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a teacher reads a thick book to the class and then asks about a small detail from a page in the middle. Some kids remembered the beginning and the end but missed the middle. Some neural networks try to keep the whole book in their head at once. It does not always work: they sometimes miss the middle too, so their answers are worth checking.
In plain words
Everything you send a model at one time, together with its answer, must fit into its working memory, the context window. A few pages used to fit there; now some models hold whole books and folders of contracts. This is convenient: you can upload the whole document and ask about it without cutting it into parts. But more does not mean more careful. On a very long input the model misses details more often and answers more slowly, and each request costs more, because you pay for all the tokens you send.
For business
Work with large documents gets easier: compare two versions of a contract, find contradictions in tender documents, review a year of correspondence with a client. According to Anthropic’s documentation, a window of 1 million tokens holds about 555,000 words with the current tokenizer. But the bill grows with the volume: if every question carries the whole archive, token costs quickly outrun the benefit. The repeated part of the input is made cheaper with caching, and for large knowledge bases searching for the right fragments is usually more cost-effective. Before adoption, test quality on documents of the length you will work with, and the response time. Such a test shows which approach is cheaper for your assistant.
For a specialist
Long context means a model works with an input from tens of thousands to millions of tokens. There are two limits: the computational cost of attention and the generalization of positional encoding beyond the lengths seen in training. To extend it, people use RoPE position interpolation, efficient attention implementations and distributed schemes such as ring attention. The Gemini 1.5 report (2024) stated near-perfect fact retrieval, above 99%, at lengths of at least 10 million tokens. However, the needle in a haystack test checks only the search for one fact. In the RULER benchmark (Hsieh and colleagues, 2024), all 17 models claimed a window of 32,000 tokens or more, but only about half kept satisfactory quality at that length, especially on tasks with multiple hops and aggregation. Hence the difference between stated and effective length. In practice a long input is combined with fragment retrieval, prefix caching and compression of the dialog history.
Example
A lawyer uploads a supply contract with all its appendices to a model and asks it to find the clauses where payment terms differ. The model names these places with section numbers, and the lawyer checks each one by hand.
Related terms
Questions and answers
Do you need RAG if the model has a long context?
Sources
- Gemini Team, Google, Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024arxiv.org
- Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models?, 2024arxiv.org
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023arxiv.org
- Anthropic, Models overview (Claude documentation, checked 16 October 2026)platform.claude.com
Where companies use it
Not sure whether your company needs this?
AI solutions map