MMOOZZGG.com

How to build an AI agent

How to evaluate AI agents: metrics and open tools

6 min read6 sources

To evaluate an AI agent is to answer one question: did the agent become more accurate after you changed the instructions, the model or the knowledge base? Anthropic advises starting with 20–50 simple tasks taken from the agent’s real mistakes: you do not need hundreds of tasks at the start.

On this page · 5 sections

A developer is needed

How an evaluation differs from a test

An evaluation (eval) is a check in which the agent gets a task and a rule or a reviewer scores its result. This is how Anthropic defines it in its guide to agent evaluation of 9 January 2026.

A test before launch answers the question “can we release it”. An evaluation runs all the time: after each change you see in numbers whether quality went up or down.

An agent is harder to evaluate than a chatbot. It takes many steps in a row, calls tools and changes data in your systems, so you look both at the final answer and at the path to it.

Anthropic divides evaluations into two kinds. Capability evaluations check what the agent can do and give a low percentage at the start. Regression evaluations check that the agent has not lost what it could do, and they should pass almost always.

The question set and forbidden answers for checks before launch are covered on the page on agent testing. Here we look at how to give scores and what tools to count them with.

Who gives the score

There are three kinds of reviewers, and each has its own cost of error.

Code. A program compares the answer with the correct one and checks the format and which tool was called. This is fast, cheap and the same on every run, but code will not understand a correct answer put in other words.

A judge model. Another model scores the answer on a scale or against a list of requirements. It is flexible and sees meaning, but its scores drift from run to run and cost money, so the judge is checked against people.

A person. An employee or an expert reads the answers and gives a score. This is the reference used to tune the judge model, but it is expensive and slow.

In practice they are combined: code checks the tool calls and the format, the judge model checks the tone and completeness of the answer, and a person reads a sample of dialogs.

Agent metrics

The main metric is the share of tasks the agent completed. It booked a customer at the right time, created a deal with the right amount, answered according to the current price list.

The second is tool call correctness. Did the agent choose the right function and pass the right parameters? The open DeepEval library has ready metrics for this: task completion, tool correctness, argument correctness, extra steps.

The third is stability. A model answers the same question in different ways, so each task is run several times. The pass^k metric shows the probability that the agent succeeds in all k attempts in a row.

An example from Anthropic’s guide: if an agent solves a task in 75% of attempts, the chance of passing three attempts in a row is about 42%. For an agent that talks to customers, this is the figure that matters, not one lucky attempt out of several.

Two more metrics: response time and the cost of a task in tokens. The business benefit is measured separately, by conversion and deals.

Open tools

For the first 20–50 tasks you can count the scores in a spreadsheet. After that, open programs that you install on your own computer or server are more convenient.

promptfoo: a command-line program and library for evaluating and attack-testing applications built on language models, MIT license. The evaluation runs locally, and the tasks and answers stay on your machine.

Langfuse: a platform for logging calls and evaluating agents, MIT license except for the ee folders. It installs on your own server with Docker Compose and supports a judge model, code checks, manual labeling and task sets.

DeepEval: an open Python evaluation library built like unit tests, Apache 2.0 license. It has the agent metrics from the section above.

Any evaluation needs tracing, that is, a log of all the agent’s steps: which tool it called, what it got and what it answered. Without it you cannot tell whether the agent or the reviewer made the mistake.

Evaluation tasks come from real conversations, and these contain customers’ names and phone numbers. Under 152-FZ, Russia’s personal data law, the data of Russian citizens is collected and stored in databases in Russia, so logs and task sets are kept on your own server or stripped of personal data. How we handle this is described on the page on data security.

How to build evaluation into your work

1. Collect 20–50 tasks from the agent’s real mistakes and write down the correct result for each.

2. Decide who checks each task: code, a judge model or a person.

3. Run the set before the change and save the scores as a baseline.

4. After each change to the instructions, the model or the knowledge base, run the set again and compare the scores.

5. Once a week, read the logs of a few dialogs in full. Anthropic advises not to trust the scores until someone has read the logs: otherwise you cannot see whether the agent or the reviewer made the mistake.

Tasks the agent solves reliably move to the regression set and run at every change. This way an old mistake does not quietly come back.

Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, partly because of unclear value. An evaluation in numbers answers the question of value before the CFO asks it; in our pilot of 14 days, the metric is written into the contract.

Questions and answers

Can you trust a score that a model gives?

Only after checking it against people. A judge model is flexible and cheaper than a person, but its scores drift from run to run. Give the same 20–50 answers to the judge and to an employee and compare: if the scores differ, rewrite the requirements for the judge.

How many tasks do you need for a first evaluation?

Anthropic advises starting with 20–50 simple tasks taken from real mistakes. At an early stage each change has a visible effect, so a small set is enough. The set grows with the agent: each new mistake becomes a new task.

Sources and fact-checking

Checked by the editors on 16 October 2026.

  1. Gartner, 25 June 2025 (gartner.com)
  2. Brace Law Firm, 9 July 2023 (brace-lf.com)
  3. Anthropic Engineering, 9 January 2026 (anthropic.com)
  4. GitHub, promptfoo/promptfoo, 2026 (github.com)
  5. GitHub, langfuse/langfuse, 2026 (github.com)
  6. GitHub, confident-ai/deepeval, 2026 (github.com)

Found a mistake? Tell us

Publications

Mike Mozg

Agentic AI use cases at AT&T: nearly 500 processes now run on agents

AT&T no longer uses AI only to help staff work faster: it is rebuilding whole business processes around agents. SiliconANGLE reported on 14 November 2026 that nearly 500 agentic AI use cases at AT&T are already in production. Here is how the company rebuilds them, where a human still checks the agent, and what other large companies can take from it.

News4 min
Contact us