How document search works
RAG (Retrieval-Augmented Generation) means the model searches first and answers second. Documents are cut in advance into chunks of a paragraph or two and turned into vectors: sets of numbers that let a program find texts close in meaning.
Vectors are kept in a vector database. It searches not by exact words but by meaning: for the question “how do I return a product” it also finds the section “returns and exchanges”.
When a customer or employee asks a question, the agent takes a few of the closest chunks and passes them to the model together with the question. The model answers from the text it found and names the document the answer came from.
To update the agent’s knowledge, replace a document in the base and process it again. The whole path, step by step, is in our guide on how to build a RAG system in 7 steps.
When you need RAG
You need RAG when the answer lives in documents that are many and change often: a support knowledge base, internal rules, a catalog with specifications, contracts, employee instructions.
If there are few documents, search may be unnecessary. Anthropic wrote in September 2024 that a knowledge base under 200,000 tokens, about 500 pages, can go into the model request whole. A token is a piece of a word that the model uses to count text volume and price.
RAG will not help where the answer comes from a live system: a free calendar slot, an order status, warehouse stock. The agent gets such data through a tool at the moment of the question; see our page about an agent without a knowledge base.
A simple test: if an employee answers the question by opening a folder of documents, RAG fits the task. If they open the CRM or 1C, the accounting system common in Russia, the agent needs access to the system.
What to build it with
A visual builder is enough for a first agent with a knowledge base. Flowise is released under the Apache 2.0 license. Dify is also on Apache 2.0, but with two conditions. Without permission, you may not run a multi-tenant cloud service like Dify on it or remove the logo from the console.
In Flowise, the knowledge base is called a Document Store. The steps from the documentation:
1. Create a Document Store and give it a name.
2. Choose a Document Loader, for example PDF Loader for PDF files, and upload the file.
3. Choose a Text Splitter, for example Recursive Character Text Splitter. Set the chunk size in the “Chunk Size” field and the overlap between neighboring chunks in the “Chunk Overlap” field.
4. Check in the preview how the text is cut into chunks, and start processing.
5. On the “Select Embeddings” card, choose the model that turns text into vectors, and on the “Select Vector Store” card, choose the vector database. The vector dimensions of the model and the database must match, or the upload fails.
6. Click “Upsert” to load the data into the base, then “Retrieval Query” to see which chunks a question finds.
7. Connect the store to the agent: in Agentflow through the Agent node, or directly through the vector database and the embedding model.
In Dify the path is shorter: the “Knowledge” section, the “Create” button and the “Create a ready-to-use knowledge base” option. Then you upload files, set up chunks with a preview and choose the indexing and search method.
Answer quality
Most often the agent fails not in the answer but in the search: it found the wrong chunk and honestly retold it. So quality is checked at two steps: was the right place found, and did the model pass it on correctly.
Known techniques raise search accuracy. In September 2024, Anthropic showed that adding a short note to each chunk about where it comes from, and searching by meaning and by words at once, cuts failed searches by 49%. With reranking of the results, the cut is 67%.
The common causes of bad answers are simpler than any technique: scans without a text layer, old document versions next to new ones, tables cut in half. Cleaning the documents usually helps more than changing the model.
Write three rules into the agent’s instructions: answer only from the chunks found, name the document, and say “I don’t know” if the base has no answer.
Before launch, collect a few dozen real questions with correct answers and run them after every change to the base. Our page on testing an AI agent explains how.
Document security
A knowledge base gives out everything loaded into it. If you put the payroll there, the agent will retell it to anyone who asks the right way.
OWASP, an open community of security specialists, names vector database weaknesses in its 2025 risk list: with weak access separation, the agent gives out chunks a user must not see. Its advice: access rights at the store level and separate datasets for different user groups.
In practice this means one base for customers and a separate one for employees. Restricted documents sit only in the base that is connected for the people allowed to see them.
If the documents contain customers’ personal data, 152-FZ, Russia’s personal data law, applies. Under Part 5 of Article 18, data of Russian citizens may not be recorded and stored in databases outside the country. This covers both the vector database and the model that reads the chunks. How we store data is described on our page on data security.