Why AI makes mistakes
A language model does not know your prices, terms, or rules until you give them. Without rules, it answers plausibly, but from its own idea of how things usually are. This is where made-up prices and promises the company never gives come from.
Six causes we see when we review agents that are already running:
1. There are no rules, or they live in one employee’s head.
2. The price list and terms are out of date: prices changed, and nobody told the agent.
3. The agent was trained on made-up examples, while customers write with typos and in three messages.
4. There is no handover to a person: the agent answers complaints and disputed cases too.
5. Nobody reads the dialogs, and the agent repeats its mistakes.
6. There is no support, and after six months all of this piles up.
These are the people and process problems from the BCG study. Each cause is fixed by rules, a price list source, and an owner, not by changing the model. The details are in the article on why an AI agent gives wrong answers.
Where a person is needed
Set the line between the agent and a person before the first customer. The agent handles standard questions where the answer is in the rules and the knowledge base: prices, terms, order status, booking. It passes everything else to an employee with the conversation history.
A person is needed where an error costs a lot or a compromise is needed:
1. Complaints and conflicts with a customer.
2. Discounts, exceptions to the rules, and non-standard terms.
3. Signing documents, payments, and negotiations.
4. Questions the rules have no answer to.
The right share of handovers to a person is not zero. An agent that hands over nothing also answers where it should have called an employee.
The process owner is responsible for the line: the head of the department decides what the agent handles alone and approves the rules. Who else is on the team and what they own is described on the page about the structure of an AI department.
Which rights to give an agent
An AI agent is a program with an account in your CRM, phone system, and messengers that takes messages from strangers. The wider its rights, the more an error or someone else’s instruction hidden in a message will cost.
There is one rule: rights only for its own process. A sales agent needs to create and update deals, write the message history, and read the knowledge base. It does not need to export the customer base or have administrator rights.
What else to check:
1. The agent’s actions are written to your CRM log under its own name.
2. The agent does not follow commands in incoming messages that change its rules; it passes such requests to a person.
3. Your administrator can revoke the agent’s access in minutes.
Why this matters more now that AI can find vulnerabilities in other people’s code is covered in the article on which rights to give an AI agent.
How to catch a mistake before the customer does
In the first weeks, the agent is learning, and there will be mistakes. What matters is that they are visible and fixed within a day, not left to pile up.
A check before launch: take real customer messages from the last week, send them to the agent, and rate each answer by three questions. Is it true, does it follow the rules, and did the agent pass to a person what it should have?
During the pilot, the owner reviews the dialogs every day and corrects the rules, which takes about half an hour a day. The agent reads prices and terms from the CRM or 1C (a common Russian accounting system), not from a file, so a price change does not turn into an error.
Repeat the same check after each change of the price list, the rules, or the model: run the same messages and compare the answers with the last time. This way, an error introduced by an edit shows up the same day, not through a complaint.
After launch, the agent’s work is measured by a number: the share of questions closed with no person, the share of handovers, and the number of corrected rules. If the number drops, it shows in the report before it shows in customer complaints.
In a 30-minute review, we will check your agent against this list, even if we did not build it.