Glossary
What is a validation set?
In short
A validation set is a held-out part of the data on which developers compare model versions and settings during development, without touching the final test.
Machine learning · validation set
MMOOZZGG.com editorsChecked: 1 min read
For a child
You are getting ready for a school concert and sing a song in different ways: louder, softer, faster. To choose, you ask your grandmother to listen. She did not learn the song with you, so she will honestly say which version sounds nicer. The real audience comes only on the day of the concert. A computer has such a grandmother too: a special set of examples.
In plain words
Developers almost never train just one model. They try different algorithms and settings and keep the one that works. It makes no sense to compare them on the data the model learned from, because it has already seen that data. So a separate portion of examples is held back for checks along the way. It decides which version to keep and when to stop training. But after many comparisons, this portion is also a little “used up”, and the final score needs one more set that nobody has touched. It is like practice exams at school: you prepare with them, but the real exam shows the result.
For business
This part of the data protects you from a model tuned to pass the check. If a contractor picks a version on the same data they later report on, the number in the report will be too high. The right scheme has three parts: training, validation for the choice and a test for the final score, and the customer keeps the test set. The more versions were tried, the more this boundary matters. One more detail: validation examples should look like real work. For a sales forecast, that means the latest months, not random days from the whole history. Ask how the data was split and how many times each part was used. Write these rules into the pilot brief.
For a specialist
A validation set (also called a development set) is held-out data for model selection: comparing architectures, tuning hyperparameters and early stopping. The scikit-learn documentation explains its purpose: if you tune parameters on the test set, knowledge about it “leaks” into the model, and the metric no longer reflects generalization; so the choice is made on validation, and the test is kept for the final check. The downside of a three-way split is that fewer examples remain for training and the result depends on a random split; with little data, cross-validation replaces a single validation part. In Keras, the EarlyStopping callback watches val_loss by default, stops training after patience epochs without progress and can restore the weights of the epoch with the best metric value (restore_best_weights). Choosing many times on the same validation set leads to overfitting on it, so with many experiments the test set stays closed until the end.
Example
A team tuned the tree depth for a churn model: it tried ten versions and scored each on a validation set of last quarter’s customers. The chosen version was then checked once on test data from the current quarter, and the result almost matched the validation score.
Related terms
Questions and answers
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map