MMOOZZGG.com

Glossary

What is a validation set?

In short

A validation set is a held-out part of the data on which developers compare model versions and settings during development, without touching the final test.

Machine learning · validation set

MMOOZZGG.com editorsChecked: 1 min read

For a child

You are getting ready for a school concert and sing a song in different ways: louder, softer, faster. To choose, you ask your grandmother to listen. She did not learn the song with you, so she will honestly say which version sounds nicer. The real audience comes only on the day of the concert. A computer has such a grandmother too: a special set of examples.

In plain words

Developers almost never train just one model. They try different algorithms and settings and keep the one that works. It makes no sense to compare them on the data the model learned from, because it has already seen that data. So a separate portion of examples is held back for checks along the way. It decides which version to keep and when to stop training. But after many comparisons, this portion is also a little “used up”, and the final score needs one more set that nobody has touched. It is like practice exams at school: you prepare with them, but the real exam shows the result.

For business

This part of the data protects you from a model tuned to pass the check. If a contractor picks a version on the same data they later report on, the number in the report will be too high. The right scheme has three parts: training, validation for the choice and a test for the final score, and the customer keeps the test set. The more versions were tried, the more this boundary matters. One more detail: validation examples should look like real work. For a sales forecast, that means the latest months, not random days from the whole history. Ask how the data was split and how many times each part was used. Write these rules into the pilot brief.

For a specialist

A validation set (also called a development set) is held-out data for model selection: comparing architectures, tuning hyperparameters and early stopping. The scikit-learn documentation explains its purpose: if you tune parameters on the test set, knowledge about it “leaks” into the model, and the metric no longer reflects generalization; so the choice is made on validation, and the test is kept for the final check. The downside of a three-way split is that fewer examples remain for training and the result depends on a random split; with little data, cross-validation replaces a single validation part. In Keras, the EarlyStopping callback watches val_loss by default, stops training after patience epochs without progress and can restore the weights of the epoch with the best metric value (restore_best_weights). Choosing many times on the same validation set leads to overfitting on it, so with many experiments the test set stays closed until the end.

Example

A team tuned the tree depth for a churn model: it tried ten versions and scored each on a validation set of last quarter’s customers. The chosen version was then checked once on test data from the current quarter, and the result almost matched the validation score.

Related terms

Questions and answers

How is a validation set different from a test set? You use the validation set to choose the model and its settings, so developers return to it many times. The test set is opened once, at the end, to get an honest score.
How big should a validation set be? Big enough that the difference between model versions does not drown in random noise. With little data, cross-validation replaces a single held-out part.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us