1. Time to first reply
The median, not the average: one email answered after three days spoils the average, while the median shows what a typical client sees. Measure it in the CRM or messenger for a month before the pilot and during the pilot. This is the agent’s first win: a minute instead of hours.
2. Share of requests closed without a person
How many dialogs the agent brought to a result by itself: answered, booked, created a deal. Count only those where the client did not write the same question again. For support, our benchmarkAI benchmark. To benchmark AI models means to compare them on one standard set of tasks with known answers. That set is called an AI benchmark. from the home page is 80% of typical questions; the pilot shows the exact share for your process.
3. Share of requests that reach the CRM
The main metric for sales. Before the agent, managers enter some requests into the CRM, and the rest live in chats. With the agent, each qualified request is a deal with a history. It is counted as the ratio of deals to requests.
4. Share of handovers to a person
Lower is not always better. Too low a share means the agent takes on what it should not; too high means the rules are not written down. In a pilot, this number usually falls by the end of the second week, and that is normal.
5. Deals and money
How many deals from requests the agent handled reached payment. It is counted last, because the deal cycle is longer than the pilot. So in the contract the pilot metric is the first four, and we look at the money a month or two later.
Which numbers lie
If you are deciding how to measure the performance of a chatbotchatbot. A chatbot is a program that exchanges messages with a person in a chat window: on a website, in a messenger or in an app. or an agent, start with what not to count. “Number of dialogs”: an agent can run thousands of useless ones. “Customer satisfaction” from a “rate this answer” button: only a few people press it. “Time saved” without the 1.3 factor and without subtracting support. Check what the contractor reports on before you sign.
How to measure before the start
Take a month of conversations and count the median reply time and the share of requests in the CRM, by hand or with an export. It takes an hour, and it is the only way to say honestly later what changed. We do this with you in the first three days of the pilot.
What the pilot report looks like
The report has three columns for each metric: the month before the pilot, the first week and the last five days. We do not count the first days, because the agent is learning. An example of the structure without numbers: median time to reply; share of requests closed without a person; share of requests in the CRM; share of handovers to a person and their reasons by group; deals from the agent’s requests on the report date. Under the report is a list of changes over two weeks: what changed in the rules and why.
The same report becomes the basis of support: the same five numbers every month, so you can see whether the agent improves or degrades.
The most common mistake
Counting the metric over all requests, including those not given to the agent. If complaints and disputed cases go to people by the rules, they should not spoil the share closed without a person. Count by the groups of requests the agent handles, and show separately how many went to people and why. That keeps the number honest and comparable from month to month.
In short
Five metrics: median time to reply, share closed without a person, share of requests in the CRM, share of handovers to a person, and deals. Measure them before the start on a month of conversations, write the first four into the contract and look at the money a month or two later. Do not trust the number of dialogs or button ratings.