MMOOZZGG.com

Glossary

What is an AI benchmark

In short

To benchmark AI models means to compare them on one standard set of tasks with known answers. That set is called an AI benchmark.

Language models · benchmark

MMOOZZGG.com editorsChecked: 1 min read

For a child

On one day, all classes in a school write the same test. The tasks are the same, and the teacher knows the answers in advance, so it is easy to see who has more correct solutions. For neural networks, people also make shared tests. But if someone has seen the tasks in advance, the score means nothing, so the tasks are kept private.

In plain words

To learn which network is stronger in math, programming or history, you give each the same set of questions with ready answers. Each model gets a percent of correct answers, and tables are built from it. This shared set is the benchmark. There are sets of school and university questions, coding tasks and tests of finding facts in a long text. It is like a car ranking by fuel use: the number is useful, but on a real trip with a load the use will be different.

For business

Public result tables help you remove weak models quickly, but they answer poorly the question of which model fits you. The tests are mostly in English and on academic tasks, while you have Russian, your own documents and your own formats. Also, tasks leak onto the internet and into training data, and the model solves them from memory. Vendors prefer to show the sets where they look good. So the decision is made on your own set of real requests with reference answers: it shows the difference in quality, price and speed. A review of which solutions and models companies actually deploy helps narrow the choice in advance.

For a specialist

A benchmark is a fixed set of tasks with reference answers and a metric for a reproducible comparison of models. Examples: MMLU (Hendrycks et al., 2020), multiple-choice questions on 57 subjects from elementary math to law; HumanEval (Chen et al., 2021), generating Python functions from a docstring, with functional correctness checked by tests; SWE-bench (Jimenez et al., 2023), 2,294 tasks from real issues in 12 Python repositories, where Claude 2 solved 1.96% at the time of publication. The main problems are test leakage into training data (data contamination), saturation, when models hit the ceiling, sensitivity to the prompt format, and fitting to a specific test. That is why closed and regularly updated sets appear, along with evaluations by a judge model or by people.

Example

A company chooses a model to answer questions on contracts. In the public tables two models are almost equal, but on a hundred of its own lawyers’ questions one answers correctly noticeably more often, and the choice is made by this test.

Related terms

Questions and answers

What is an AI benchmark? It is a shared set of tasks with correct answers on which different models are tested. The result is the percent of correct answers, so the models can be compared with each other.
Can you trust AI rankings? As a guide, yes; as the basis for a choice, no. The tasks may have reached the training data, and your tasks differ from the test ones.

Test the models on your own examples.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us