MMOOZZGG.com

Glossary

What is a long context LLM?

In short

Long context is the ability of a language model to take in and use a very large amount of text in one request: thick documents, a long correspondence or a whole code repository.

Language models · long context

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine a teacher reads a thick book to the class and then asks about a small detail from a page in the middle. Some kids remembered the beginning and the end but missed the middle. Some neural networks try to keep the whole book in their head at once. It does not always work: they sometimes miss the middle too, so their answers are worth checking.

In plain words

Everything you send a model at one time, together with its answer, must fit into its working memory, the context window. A few pages used to fit there; now some models hold whole books and folders of contracts. This is convenient: you can upload the whole document and ask about it without cutting it into parts. But more does not mean more careful. On a very long input the model misses details more often and answers more slowly, and each request costs more, because you pay for all the tokens you send.

For business

Work with large documents gets easier: compare two versions of a contract, find contradictions in tender documents, review a year of correspondence with a client. According to Anthropic’s documentation, a window of 1 million tokens holds about 555,000 words with the current tokenizer. But the bill grows with the volume: if every question carries the whole archive, token costs quickly outrun the benefit. The repeated part of the input is made cheaper with caching, and for large knowledge bases searching for the right fragments is usually more cost-effective. Before adoption, test quality on documents of the length you will work with, and the response time. Such a test shows which approach is cheaper for your assistant.

For a specialist

Long context means a model works with an input from tens of thousands to millions of tokens. There are two limits: the computational cost of attention and the generalization of positional encoding beyond the lengths seen in training. To extend it, people use RoPE position interpolation, efficient attention implementations and distributed schemes such as ring attention. The Gemini 1.5 report (2024) stated near-perfect fact retrieval, above 99%, at lengths of at least 10 million tokens. However, the needle in a haystack test checks only the search for one fact. In the RULER benchmark (Hsieh and colleagues, 2024), all 17 models claimed a window of 32,000 tokens or more, but only about half kept satisfactory quality at that length, especially on tasks with multiple hops and aggregation. Hence the difference between stated and effective length. In practice a long input is combined with fragment retrieval, prefix caching and compression of the dialog history.

Example

A lawyer uploads a supply contract with all its appendices to a model and asks it to find the clauses where payment terms differ. The model names these places with section numbers, and the lawyer checks each one by hand.

Related terms

Questions and answers

Do you need RAG if the model has a long context? For one large document a long window is often enough. For a knowledge base of thousands of documents, searching for the right fragments stays cheaper and faster and also shows the source of the answer.

Many systems combine both approaches.

Why does a model lose details in a long text? Research shows that models use information from the middle of a long input worse than from its beginning and end. The stated window length does not mean equal quality across the whole length, so test it on your own documents.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us