Glossary
What are hyperparameters?
In short
Hyperparameters are training settings that a person sets before the start and that the model does not change itself: the learning rate, the batch size, the number of epochs and the size of the network.
Machine learning · hyperparameters
MMOOZZGG.com editorsChecked: 1 min read
For a child
When you bake a pie with your mom, you decide in advance which pan to use, how hot to make the oven and how many minutes to wait. The pie in the oven does not decide this. If you choose badly, it burns or stays raw. Before a computer learns, there are decisions like these too, and people make them. They are called hyperparameters.
In plain words
A model has two kinds of numbers. The first it finds itself during training: these are its weights. The second a person sets in advance, and they control the process itself: how fast to learn, how many examples to take at a time, how many passes to make over the data and how many layers the network will have. It is hard to guess them on the first try, so several options are tried and the results are compared on validation data. It is like a soup recipe: the ingredients are the same, but the taste depends on how long you cook it and when you add salt.
For business
Hyperparameter tuning takes a large part of the training budget: each set of settings means a separate run on graphics cards. If a contractor tries twenty combinations, you pay for twenty trainings, although only one goes into use. So ask in advance how many runs the estimate includes, how they are chosen and on which data they are compared. There is also a quality risk: if the settings were fitted by looking at the test set many times, the model will do worse on real clients than promised. Require a held-out set that the team did not touch until the final check, and recorded settings, so training can be repeated without the original team. Such terms are easy to fix in the pilot specification.
For a specialist
Hyperparameters are parameters of the training process and architecture that are not optimized by the gradient of the loss function: the learning rate, batch size, number of epochs, regularization coefficients, dropout rate, number and width of layers, and optimizer parameters. In scikit-learn terms, they are the arguments of an estimator’s constructor, such as max_depth of a random forest. Hyperparameter optimization is guided by quality on a validation set or by cross-validation. Grid search scales poorly to a high-dimensional space: Bergstra and Bengio (JMLR, 2012) showed that random search finds models as good as the grid at a small fraction of the computing. The next step is Bayesian optimization: Snoek, Larochelle and Adams (2012) modeled quality with a Gaussian process and chose the next point from its posterior distribution. A separate group of methods drops weak runs early (successive halving, Hyperband). The final score is taken on a test set that did not take part in tuning.
Example
A retail chain ordered a customer churn forecast model. The contractor tried 40 combinations of tree depth and learning rate and chose the option that gave fewer false alarms on held-out data for the last quarter.
Related terms
Questions and answers
Sources
- Bergstra, Bengio, Random Search for Hyper-Parameter Optimization, JMLR, 2012jmlr.org
- Snoek, Larochelle, Adams, Practical Bayesian Optimization of Machine Learning Algorithms, 2012arxiv.org
- scikit-learn, Glossary of Common Terms: parameter, official documentation, checked 16 October 2026scikit-learn.org
Not sure whether your company needs this?
AI solutions map