Glossary
What is a test set in machine learning?
In short
A test set in machine learning is data that a model has not seen in training or in tuning. You use it once to get the final quality score.
Machine learning · test set
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine you learned to ride a bike for a long time in your own yard, where you know every bump. Then you and your dad go to a park you have never seen. Only there can you tell whether you really know how to ride or just remember your yard. A computer is also checked on unfamiliar examples that were hidden from it in advance, and these examples are called the test set.
In plain words
When a model is ready, you need to know how it will act on new customers, emails or images. So you set aside part of the data in advance and show it neither to the model nor to the people who tune it. At the end, the model answers these examples once, and its answers are compared with the correct ones. That gives an honest score. If someone peeks at the hidden data and adjusts the model to it, the score comes out too high. It is like exam papers kept in a sealed envelope until exam day.
For business
The number on this data is the one that should go into the contract and the acceptance report. A reliable setup: you select and keep the test examples yourself, for example last month’s requests, and give them to the contractor only for the final run. That way nobody can tune the model to the check, even by accident. The risks: the test does not look like real work (only easy cases, a different period), it is too small, so a couple of lucky answers change the percentage, or it was used many times during development. Researchers who rebuilt the test for a well-known image dataset saw model accuracy drop by 11–14%. So after launch, quality is also checked on the live flow. How to fix such a check in the contract is decided while preparing the pilot.
For a specialist
A test set (holdout set) is data held back for an unbiased estimate of the generalization error of a model that has already been chosen. It takes no part in training, in hyperparameter tuning or in choosing the architecture: any decision made on it turns it into a validation set and makes the estimate optimistic. Recht et al. (2019) repeated the procedure for collecting the CIFAR-10 and ImageNet test sets and saw accuracy drop by 3–15% and 11–14% respectively. The authors link the drop not to repeated use of the test but to models transferring worse to slightly harder images. Hence the requirements: the test comes from the target distribution, is large enough for narrow confidence intervals, does not overlap with training by objects and groups, and for time-based data lies later than the training data. Public benchmarks suffer from contamination, when test examples end up in the training corpora of language models.
Example
A clinic chain accepted a model that transcribes doctors’ recordings only after a run on two hundred fresh recordings. The chief physician selected them and did not show them to the contractor. On them, the error rate was higher than in the sales presentation, and the launch was postponed until the model was improved.
Related terms
Questions and answers
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map