Why the effect is not visible
According to the MIT NANDA 2025 report, 95% of organizations got no measurable return from generative AI. The key word is “measurable”: without a measurement, even working AI looks like a cost.
A typical picture: the system is launched, staff say work is easier, and the CFO sees no change in the reports. Six months later the project is closed as too expensive.
The first reason: success was measured by the fact that the system was launched. A launch tells you about the contractor’s work, not about the company’s money.
The second reason: the metric was chosen after the start. Then there is no “before” point, and any number can be explained by the season, a new manager or advertising.
The third reason: too many metrics. Nobody reads a report of twenty indicators, and nobody makes decisions from it.
One metric in money
One metric per process. If there are five metrics, one will grow, another will fall, and the argument will never end.
The metric must be counted in your system without manual counting: in the CRM, the booking system, the phone system. Otherwise people will dispute the number.
The metric must depend on AI. Company revenue depends on the season, prices and advertising, so a pilot does not measure it. Response time to a lead depends on the agent directly.
Examples for typical processes. Leads: time to first reply and the share of leads entered in the CRM; booking: the share of requests that ended in a booking without an administrator; support: the share of questions closed without an operator.
Each such metric is turned into money. Response time affects the share of leads that reach a deal. Questions closed without an operator turn into staff hours, and the hours are multiplied by the salary with taxes.
The article on calculating the effect explains how to estimate the effect in money for each solution, together with the investment and the payback period.
The metric in the contract
A metric in a presentation commits nobody to anything. A metric in the contract commits both sides: the contractor knows what it answers for, and you know what you accept.
The contract records three things: which number we measure, its value now and in which system it is counted. The “now” value is measured before the start, in your own system.
The check date also goes into the contract. Without a date the check keeps moving, and the project keeps getting money.
This is how our pilot works: one AI agent on one process, from $1,499, 14 days, with the metric in the contract. What it includes and how we choose the metric is in the article on the AI agent pilot.
If a contractor refuses to write down the metric, that is a signal: it is not sure of the result or does not understand your process.
A report with three numbers
At the end of the pilot you get a report with three numbers: the metric before the start, the metric in the last days of the pilot, when the agent is already trained, and the share of cases the agent passed to a person.
The first days of the pilot do not count: the agent is learning your rules and conversations, and the number at the start is too low.
The third number matters as much as the first two. If the agent passes most requests to people, the metric may grow thanks to people, not AI. The hand-off share shows how much work really went to the agent.
The chief executive and the CFO read the report, so its numbers are in money and percentages, with no technical terms.
With three numbers, the decision takes one conversation: keep the agent, extend it to other processes or stop. After that, the company’s AI department tracks these metrics; the page on AI implementation KPIs describes how this works.
What to do if the number did not grow
A failed pilot with a metric is more useful than a successful one without it: you know why it did not work, and you spent 14 days on it, not a year.
You see the cause in the agent’s conversations. Usually there are three: the agent lacked information for its answers, the process was not what it was described as, or staff did not pass requests to the agent.
If information was missing, the knowledge base and the rules are extended. If the process is different, you choose a process with bigger losses and a simpler metric. If people bypass the agent, the question goes to the owner and to training, not to the technology.
Do not extend the pilot in the hope that the number will grow by itself. If the metric did not move during the pilot, an extension usually just delays the same conclusion.
If there is no payback, we write that in the report. After the pilot there are three options: stop, keep the agent on support from $480 a month, which you can cancel in any month, or build an AI department of 3–5 agents.