MMOOZZGG.com

Glossary

What is chunking in RAG?

In short

Chunking is cutting documents into small pieces (chunks) that a RAG system then searches.

Search and RAG · chunking

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine a thick book of fairy tales. If a friend asks the name of the cat in one tale, you will not reread the whole book; you will open the right chapter. And if the book is split in advance into small chapters with bookmarks, finding the answer is even easier. Chunking is this kind of splitting of a big book into handy pieces for a computer.

In plain words

A model cannot read a whole folder of documents at once, and search works badly with whole files of a hundred pages. So texts are cut in advance into pieces called chunks. You can cut by number of words, by paragraph, or by document section. Each piece is stored separately, and for a question the system finds several that fit. Size matters. A piece that is too small loses meaning: the phrase “the term is ten days” means nothing without its heading. A piece that is too big pulls extra text into the answer.

For business

Chunking in RAG largely decides whether the assistant finds the right contract clause or returns the next one. Mistakes here are cheap at the start and expensive later: after launch, rebuilding the index means new embedding costs and another quality check. Tables, scans, price lists, and documents with running clause numbers are the hardest: they are cut by their own rules, not by character count. A useful practice is to add the document name, the section, and the version date to each chunk. Before you choose a method, look at which documents you have and what state they are in. This part is reviewed together with data preparation for AI.

For a specialist

Chunking is the indexing step where a document is split into retrieval units. The basic strategies are fixed size in tokens or words with overlap, recursive splitting by separators (sections, paragraphs, sentences), splitting by document structure (headings, tables), and semantic splitting, which sets a border where the topic changes sharply between neighboring sentences. In Dense Passage Retrieval (Karpukhin et al., 2020), Wikipedia articles were split into non-overlapping blocks of 100 words. The main problem of independent chunking is that a chunk loses the context of the parts around it. There are two ways to solve it. Contextual Retrieval (Anthropic, 2024) adds to each chunk a 50–100 token note that a model writes from the whole document. Late chunking (Günther et al., 2024) first runs the whole text through a long-context embedding model and splits the token representations only before mean pooling.

Example

In a set of rules, the clause “Review period: five working days” ended up in a separate chunk with no heading, and the assistant did not know which process it was about. After the section name was added to each chunk, the answers became accurate.

Related terms

Questions and answers

What chunk size should you choose? There is no single answer: the size is chosen for the type of documents and tested on real questions. A chunk should make sense without its neighbors but should not pull extra topics into the answer.
What is overlap in chunking? Overlap means that neighboring chunks partly repeat each other at the border. This way a thought cut in the middle lands in full in at least one chunk.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us