Glossary
What is an AI benchmark
In short
To benchmark AI models means to compare them on one standard set of tasks with known answers. That set is called an AI benchmark.
Language models · benchmark
MMOOZZGG.com editorsChecked: 1 min read
For a child
On one day, all classes in a school write the same test. The tasks are the same, and the teacher knows the answers in advance, so it is easy to see who has more correct solutions. For neural networks, people also make shared tests. But if someone has seen the tasks in advance, the score means nothing, so the tasks are kept private.
In plain words
To learn which network is stronger in math, programming or history, you give each the same set of questions with ready answers. Each model gets a percent of correct answers, and tables are built from it. This shared set is the benchmark. There are sets of school and university questions, coding tasks and tests of finding facts in a long text. It is like a car ranking by fuel use: the number is useful, but on a real trip with a load the use will be different.
For business
Public result tables help you remove weak models quickly, but they answer poorly the question of which model fits you. The tests are mostly in English and on academic tasks, while you have Russian, your own documents and your own formats. Also, tasks leak onto the internet and into training data, and the model solves them from memory. Vendors prefer to show the sets where they look good. So the decision is made on your own set of real requests with reference answers: it shows the difference in quality, price and speed. A review of which solutions and models companies actually deploy helps narrow the choice in advance.
For a specialist
A benchmark is a fixed set of tasks with reference answers and a metric for a reproducible comparison of models. Examples: MMLU (Hendrycks et al., 2020), multiple-choice questions on 57 subjects from elementary math to law; HumanEval (Chen et al., 2021), generating Python functions from a docstring, with functional correctness checked by tests; SWE-bench (Jimenez et al., 2023), 2,294 tasks from real issues in 12 Python repositories, where Claude 2 solved 1.96% at the time of publication. The main problems are test leakage into training data (data contamination), saturation, when models hit the ceiling, sensitivity to the prompt format, and fitting to a specific test. That is why closed and regularly updated sets appear, along with evaluations by a judge model or by people.
Example
A company chooses a model to answer questions on contracts. In the public tables two models are almost equal, but on a hundred of its own lawyers’ questions one answers correctly noticeably more often, and the choice is made by this test.
Related terms
Questions and answers
Can you trust AI rankings?
Sources
Not sure whether your company needs this?
AI solutions map