A pilot on one process
A pilot is a test of the agent on one working process, with real customers and an end date. One process is needed to see what exactly gave the result: the agent or something else.
Take a process with a large flow of similar requests: request intake, client booking, routine support questions. The more requests a day, the faster the statistics build up.
The agent is connected from day one to the system where the process lives: the CRM, the booking system or a messenger. Without that connection it is a demo, not a pilot.
On your side you need one responsible person: they give access, sample chats and answer rules, and they check disputed cases every day. Without such a person the agent learns from the guesses of whoever built it.
In June 2025 Gartner predicted that more than 40% of agentic AI projects will be canceled by the end of 2027 because of costs, unclear value and weak risk control. A short pilot with a price and a term answers the questions about cost and value before large investments.
Our pilot takes 14 days and costs from $1,499, and the metric is written into the contract. What the pilot includes and what you get at the end is described in the article on the 14-day pilot.
Metric
A metric is one number that shows whether the agent copes with the process. It is measured in your system without manual counting, before launch and after.
Anthropic’s developer documentation describes a good success criterion by four properties: specific, measurable, achievable and relevant to the task. “The agent answers well” fits none of them, and “the share of requests entered into the CRM” fits all of them.
Examples of launch metrics: the time to the first reply to a customer, the share of requests closed without an operator, the share of bookings without an administrator. One main metric goes into the contract, and the rest go into the weekly report.
In the same place Anthropic advises evaluating a system by several criteria at once. For an agent, quality control is added to the main metric: the share of correct answers on the test set of questions, and the answers the agent must not give.
The goal is set from the baseline number: cut the time to the first reply, or raise the share of requests without an operator, by an amount named before the start. A goal invented after launch gets adjusted to the result.
Measure the metric before the start. Without a baseline, in 14 days you will have an opinion, not a result.
Weekly review
After launch the agent talks to real customers, and its mistakes show only in the log. The log is a record of every step: the question, the tool called, the answer, the handoff to a person.
Build tools keep such a log by themselves. In OpenAI’s Agents SDK, tracing, a record of model calls, tools, handoffs between agents and checks, is on by default. It is used to find errors and watch the agent both during development and in production.
Once a week the responsible person on your side and the person who built the agent go through five steps:
1. Compare the week’s metric with the previous week and with the measurement before launch.
2. Read all dialogs with a customer complaint and all handoffs to a person.
3. List the answers where the agent got a fact, a price or a date wrong, and find the cause: the instruction, the knowledge base or the CRM data.
4. Fix the cause and run the test set again, so the fix does not break other answers.
5. Record the changes with a date, so a failed fix can be rolled back.
During the pilot the review runs every day: the agent meets questions that were not in the examples. After the first 30 days once a week is enough, if the metric holds.
When to expand
Expand the agent when the main metric has reached the goal and held for several weeks in a row, and the share of handoffs to people is not growing. If the metric did not move, first find the cause in the dialogs: the wrong process, too little data, or staff did not pass requests to the agent.
Expand one step at a time: a new channel, a new type of question or a second process. Each step gets its own metric and its own measurement before launch; otherwise it is again unclear what worked.
Stopping a pilot is also a normal outcome. If by the end of the term the metric has not moved and the cause is the process, not the setup, the money goes to the next process in the queue instead of extending the experiment.
MIT NANDA writes that buying from specialized vendors and partnerships succeed in about 67% of cases, and in-house builds a third as often. Before expanding, decide who will lead the new processes: your own team or a partner that has already brought agents to results.
Our next step after the pilot is an AI department of 3–5 agents in 4–6 weeks from $4,999, and the cost of the pilot counts toward it. Support after launch costs from $480 a month and includes reviews and fixes.