MMOOZZGG.com

Glossary

What is semantic chunking

In short

Semantic chunking is a way to cut documents into chunks by meaning. The boundary is placed where the text moves to a new topic, not after a fixed number of words.

Search and RAG · semantic chunking

MMOOZZGG.com editorsChecked: 1 min read

For a child

You are splitting a book of stories into parts for your friends. You can cut exactly every ten pages, and then one story gets torn in half. Or you can look for where one story ends and the next begins, and cut there. Then each friend gets a whole story with a beginning and an end. Semantic chunking splits texts for a computer the second way.

In plain words

Before a search, documents are cut into pieces. The simple way cuts text after an equal number of words and can break a thought in the middle. Semantic chunking is more careful. Each sentence is turned into numbers that describe its meaning and compared with the next one. While neighboring sentences are similar, they go into one piece. When the similarity drops sharply, a new topic has started, and the boundary goes there. For example, in a contract the section on payment and the section on penalties go into different chunks, even if the first is short and the second is long.

For business

The method promises more complete chunks, and so more accurate answers from an assistant. But it has a cost: each sentence must go through an embedding model, so preparing the knowledge base takes more money and time. A 2024 study showed that this cost does not pay off with a stable gain in quality compared with fixed-size chunks. So decide by a test: take your own documents, cut them both ways and compare the share of correct chunks found. For contracts and policies with a clear structure, cutting by sections and clauses is often enough. If you have very few documents, you can skip chunking and use CAG. Comparing the methods is part of preparing data for AI.

For a specialist

Semantic chunking sets chunk boundaries by the semantic similarity of neighboring sentences. A typical setup: the text is split into sentences, and an embedding model encodes each one, sometimes together with its neighbors. Then the cosine distance between consecutive vectors is computed. A boundary goes where the distance exceeds a threshold, usually set as a percentile of the distances in the document. In the LlamaIndex documentation example this is the 95th percentile, and the documentation credits the idea to Greg Kamradt. Variants include a threshold on the absolute similarity value, sentence clustering and splitting with a language model. Qu, Tu and Bao (2024) tested the method on three tasks: document retrieval, evidence retrieval and answer generation from what was found. The authors concluded that the computing cost is not justified by a stable gain over a fixed size. Other answers to the loss of context at boundaries are late chunking and adding a note about the whole document to each chunk.

Example

Lawyers load supply contracts into an assistant. With chunking by meaning, the whole section on penalties goes into one chunk. The assistant does not lose a condition that a cut by word count would have put into the next piece.

Related terms

Questions and answers

How is semantic chunking different from ordinary chunking? Ordinary chunking cuts text after an equal number of words or tokens. Semantic chunking puts the boundary where the topic changes, so the chunks have different lengths.
Should you use semantic chunking? Only after a test on your own documents. A 2024 study found no stable gain over a fixed size, and preparing the knowledge base costs more.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us